Software Quality

When I try to code, I always ask myself what’s right and what’s wrong about software quality. Sometimes, those questions aren’t easy to answer, but as software developers, we must answer them. Over my short time (4 years) as a developer, I developed certain universal and basic interrogations. I found some by reading online and others by questioning myself. When answered correctly, they can give you a hint at the quality of a software.

Every time I code, I answer all of them. Sometimes, the answer can be, for that particular project: “I don’t care”. Sometimes, for that specific line of code, I don’t have to care because there already is a structure, a framework, or a way of doing things that already makes sure the software is great. The important thing is to be aware of those questions, and to take a decision for each of them.

Those questions aren’t in any particular order. I try to split them by software project phase: design, implementation, tests, build, deployment, and monitoring. Some are useful for more than one phase. For each one of them, I try to answer it briefly and offer some potential solutions. Obviously, those solutions are based on my personal knowledge and may – and I hope so – change over time.

Design

The design phase is where everything is built. Developers must make sure they focus on the problem to solve with the right challenge in mind. Try to divide and conquer by fixing one problem at a time. At this point, it is very easy to over design something. Keep in mind the final goal and review it with peers.

Compatibility

When creating a software, think about the future.

• How will it run with other APIs?
• Can you break the API contract?
• If not, how can you change it?

Most of the time, you must make sure everything stays compatible with your API clients. There aren’t many solutions to that challenge. One of the best ways is to use API versioning. Each time there’s a breaking change on the client side, you change version. Over a long period of time, you can deprecate some API versions and then proceed to clean up some code.

Extensibility

• Can a developer easily change or add functionalities to that component?
• Should a client be able to change its behavior?

When designing a software, try to think about those different needs. You can challenge them to make them simpler. Sometimes, the requirements are per client, per line of business, or they won’t change at all. Plan ahead and figure out a way to fill all needs. Sometimes, you must let clients code what they want themselves. When they do, design a framework instead of a product. Overall, keep in mind how someone, internal or external, can add functionalities and whom they target.

Upgradability

Upgradability is a bit different than extensibility. In extensibility, you make sure you can add features and modify component behavior based on needs. When talking about upgradability, you need to make sure that you can change the whole component as the application evolves. A component of your software should not be completely bound to another system. You should ask yourself these kinds of questions:

• Can we change the database technology easily?
• How can we change the whole back-end without affecting the clients?
• Can we change an application layer without major impacts everywhere?

Across-the-board, it’s a good practice to have a module with a single purpose, which communicates with other parts of the application using a well-known standard. If you don’t, you’ll end up with a house of cards that breaks when anyone tries to change a part of it. By making sure all your modules talk to each other using a standard, you’ll be able to upgrade them separately way easier.

Some standard ways to help your modules communicate are REST or JSON_RPC and not COM Objects, binary serialized objects, or Microsoft-Only thingies (I know they are getting better). The later ones aren’t cross-platform or cross-language and they bind the module to a specific technology.

If you really don’t want to use a web server or need more speed, try something like Google Protocol Buffer or Thrift.

Debuggable

• How can I find a bug in the app?
• How can I troubleshoot a problem in a production environment?
• How can I look at variable values?

At any time, you must be able to debug an application execution. There will be bugs; make sure your software isn’t a big black box. To do so, you can add internal and external logs everywhere – it will help you understand the app flow. Logging everything all the time can consume resources and slow down your app. Having intelligent logs and error messages is tremendously important here. A good log can solve problems before going to the support team (An error happened. vs The communication failed because XYZ wasn't configured properly.). If you need more logs, you can use dynamic switches that will trace your app in specific scenarios without restarting the application. If everything else fails, make sure you can attach to the process or dump the memory to analyze it.

Implementation

Now that we’ve asked ourselves some questions about the design, we are ready to implement it. Obviously, there are still some interrogations to be answered.

Best Practices

• Am I the first one to ever solve that problem?
• What is the state of the art about that?
• How does the industry do these things?

Those are big questions that hide a lot of things. Best practices are what you learn at school or work – how to reuse code, use designs patterns, write comments, etc. (Here’s a great book about it) I’ve put them all in the same category because the end result is the same – learn and search for outside information. There’s always a good reason behind a community choice. Why do we write comments? To share knowledge. Why do we reuse code? Because it’s easier to maintain and understand. You see the picture.

In general, when implementing a solution, try to be lazy and search for existing solutions or parts of solutions. In a world that moves so fast, you can’t objectively go your own way and ignore a whole community.

Code Review

• Should I have my code reviewed by someone else?
• Can others learn from what I’ve done?
• Should I enforce a code standard?

Code review is done first for quality purposes, but it is an excellent way to learn and share information to the whole team. Over the years, I’ve learned as much as a reviewer than a reviewee. To start adopting a code review process in your team, there are many tools or processes you can use. You can enforce pull requests on your master branch or use softwares like:

• Phabricator
• Gerrit
• Reviewboard
• Crucible

Maintainability

• Is it a quick demo?
• Will it be in a product?
• How many end users will use it?
• How long will it be used?
• Does it need documentation?

A quick one-time demo that will be put to trash at the end of the week doesn’t need the same maintenance effort as a feature in the main product. A great quote about maintainability is “Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.”

On another subject, a group of researchers identified some key factors that can affect the maintenance of software. As positive impact factors, they talk about Maintenance specialists, High staff experience, Table-driven variables and data, and Low complexity of base code. They also list negative factors like: Error prone modules, Embedded variables and data, Staff inexperience, and High complexity of base code. In other words, make sure to follow best practices, keep it simple and stupid, and have a way to measure your application with tools (complexity analysis, unit testing, etc.). (Here’s another good book)

Tests

• What are the use cases?
• What could fail?
• What behavior could change?
• Do I really want to do this manually each time there’s a release?
• What should my test cover?

Try to find that pesky bug that someone would run into in production. Unit testing reduces your maintenance and saves money – it’s a medium-term investment. A bug found early in the software lifecycle costs a lot less than a bug found by a client.

Every language has its own way of doing unit tests. Take advantage of that fact and integrate unit tests in your build process. Try testing the modules individually and together. To do that, you can use mocking framework Mockito, Moq, JsMockito) to really corner a part of your application and test it easily. Those tests should be lightning fast and run each time someone makes a change. Also, for integration tests, you must mimic the real thing as much as you can. In those cases, you can use external calls, events recording/replaying, or even a real user session emulation (like Selenium or PhantomJS).

An important part of unit testing is making sure a failing test is investigated – a green build must be a team priority. All those tests will make sure that when you change something in your code, it doesn’t break anything. Running the tests systematically will accelerate development times and enable confidence in minor changes.

Build & Deploy

I’ve put together the build and deploy process because they have a lot of common goals.

Replicable

• Can anyone replicate the same setup easily?
• Are all environments essentially the same?
• Are the developers using something that looks like the production environment?

A setup that looks like the real thing really helps the developers to code and test potential pitfalls. There are a lot of tools that can help you standardize the build and deploy processes. To help you, you can check some tools like gulp, msbuild, and maven for the build process, and npm, pip, and NuGet for your dependencies management. Typically, each language has its own way of doing things.

When you know what your build process looks like, make it work into a continuous integration platform like Jenkins, Travis or TeamCity. Those tools will build your project and make sure everyone can build it correctly and not only On Their Machine^TM.

Overall, the processes must be so easy that any developer would be able to build and deploy to their dev environment. One person must not have the exclusive knowledge on how to build or deploy – there shouldn’t have any human single point of failure for these processes.

Uptime

• When an error happens, can we recover?
• Will data be lost?
• What uptime (SLA) are we aiming for?

When maintaining software, uptime and data integrity are normally very important things and there are a lot of articles that talk about them. To achieve uptime at Coveo, we usually put the upgraded module in read-only for a certain amount of time and create a parallel module that isn’t active for the clients. Then, we test it and make sure everything works correctly on the new one. When ready, we swap (gradually, when we can) the two modules. At any point, we can rollback, and it won’t affect any of our clients.

To achieve data integrity, there aren’t a lot of things we can do that scales for big volumes. The best thing I know is to release a change without making the actual breaking change – e.g., removing a database column – in the same release as the upgraded solution. After that release, the software should not use the deleted resource. After a certain number of releases, when everything falls back on the new solution, you can make the breaking change. This way, you can still rollback and make sure data isn’t lost.

Monitoring

Last but not least – the monitoring. Any quality software must have a way to monitor itself so it can improve and fix problems before they arise. Monitoring is a way for your app to communicate to developers – by communicating through logs or alerts, a developer can know if the software is happy or needs something. Don’t mute your app; give it a voice!

Measurable

• What is currently happening in my application?
• What are the performances?
• Where should I put my efforts to optimize the software?

There’s a saying in the manager world: “If you can’t measure it, you can’t manage it.” This is also true for any software you write. If someone writes a black box, they can only control the input and monitor the output. As any black box, there are millions and millions of possibilities of what could go wrong.

To give an app a voice, you can use just about anything. Make it blink, beep, or vibrate; it doesn’t really matter – simply make sure you can communicate with it in real time. Obviously, there are solutions better than others (blinking isn’t very convenient). A great way to communicate with your app is by using a log platform (Sentry, Stackdriver, Logstash, etc.) and integrate it with a logger (Log4XYZ, console logs, etc.) This way, you’ll be able to see what’s happening nearly in real time and can produce statistics out of it – it will help tremendously.

When the log part is done, you can use software like Graphite or InfluxDB to visualize and understand your software. Those tools can graph real-time your app performance and issues based on various data.

Actionable

Having your app talk to you is great, but you also have to make sure that when it’s yelling, someone is answering the call. So make sure you have a way to detect errors that create an action item somewhere. Again, there are some platforms that can alert you when something strange happens (Pager duty, VictorOps, IFTTT). Monitoring that doesn’t trigger any real action is way less effective.

This is it! These are all the questions I keep asking myself when I write and design any software. There is a lot more content we could write for each of these sections, but I’ve tried to keep it short and only give you a grasp of how I see things. Any feedback is welcome :).

Source: source.coveo.com

Using request objects with Feign

We recently decided to move our functional tests stack from python to Java, mainly to make coding them easier (our project’s backend is coded in Java) and thus increase the number of tests getting written. We needed a few things to make this possible and one of them was a complete and comprehensive Java client for the Usage Analytics API. Since a lot of the Java API clients we use internaly are built with Netflix’s Feign, I decided to give it a go.

After playing with Feign a little, I started to really like the tool. Writing an HTTP client with it is pretty easy, and it would not be a lot of work to maintain. I only had one major concern: there was no out of the box support for request objects.

Using request objects is a simple pattern that help maintain methods with many optional parameters, which is the case for some our API’s methods. Without request objects, calling a method would look like this:

statsApi.getCombinedData(from,
                         to,
                         dimensions,
                         metrics,
                         null,
                         null,
                         null,
                         null,
                         true,
                         null,
                         null,
                         null);

Not looking so good, right? Using a request object transforms the method call into this:

statsApi.getCombinedData(new GetCombinedDataRequest(from,
                                                    to,
                                                    dimensions,
                                                    metrics)
                         .withIncludeMetadata(true));

Way better!

For the request objects, we settled for a constructor that would take the required parameters of the API call in arguments. The optional parameters can then be added to the request with setters or via the fluent interface pattern.

So, this is all very nice, but it does not fix my initial concern with Feign. I have some really nice request objects, but I cannot use any of them, as they are not supported. But, since Feign is very easily extendable, I simply added support for the request objects via a homemade encoder. And thus, the ReflectionEncoder was born.

public class ReflectionEncoder implements Encoder
{
    private static final String DEFAULT_DATETIME_FORMAT = "yyyy-MM-dd'T'HH:mm:ss.SSSZZ";
    private ObjectMapper mapper;
    private Encoder fallbackEncoder;

    public ReflectionEncoder(ObjectMapper mapper,
                             Encoder fallbackEncoder)
    {
        this.mapper = mapper;
        this.fallbackEncoder = fallbackEncoder;
    }

    @Override
    public void encode(Object parametersObject, Type bodyType, RequestTemplate template)
      throws EncodeException {
      if (Request.class.isAssignableFrom(parametersObject.getClass())) {
        Map<String, Object> params = new HashMap<>();
        try {
          // Prepare the requestTemplate
          for (Method method : parametersObject.getClass().getMethods()) {
            if (method.isAnnotationPresent(QueryParam.class)) {
              String key = method.getAnnotation(QueryParam.class).value();
              Object value = method.invoke(parametersObject);
              if (value != null) {
                if (Collection.class.isAssignableFrom(method.getReturnType())) {
                  value =
                      ((Collection<?>) value)
                          .stream()
                          .map(v -> String.valueOf(v))
                          .collect(Collectors.toList());
                } else if (DateTime.class.isAssignableFrom(method.getReturnType())) {
                  value = ((DateTime) value).toString(DEFAULT_DATETIME_FORMAT);
                } else {
                  value = String.valueOf(value);
                }
                params.put(key, value);
                template.query(key, keyToTemplate(key));
              }
            } else if (method.isAnnotationPresent(PathParam.class)) {
              String key = method.getAnnotation(PathParam.class).value();
              Object value = method.invoke(parametersObject);
              if (value != null) {
                params.put(key, String.valueOf(value));
              }
            } else if (method.isAnnotationPresent(BodyParam.class)) {
              template.body(mapper.writeValueAsString(method.invoke(parametersObject)));
            }
          }

          // Replace templates with actual values
          template.resolve(params);
        } catch (
            IllegalAccessException
            | IllegalArgumentException
            | InvocationTargetException
            | JsonProcessingException e) {
          throw new EncodeException("Could not encode parameter object correctly", e);
        }
      } else {
        fallbackEncoder.encode(parametersObject, bodyType, template);
      }
    }

    private String keyToTemplate(String key)
    {
        return "{" + key + "}";
    }
}

It may look complicated, but it’s in fact pretty simple. Here is how it works: if the object received by the encoder is of the right type, it will use reflection to find the getters of the object, and depending on the annotation, inject the parameter at the right place in the RequestTemplate. Otherwise, it will use a fallback encoder.

Now, simply set the ReflectionEncoder in your client class with the builder provided by Feign and you are ready to go!

Here is a complete example of a simple client using request objects.

public interface CustomDimensionsApi extends ClientFactory.Api
{
    @RequestLine("PUT /" + ApiVersion.VERSION + "/dimensions/custom/{apiName}")
    DimensionResponse editDimension(EditDimensionRequest request);
}

public class EditDimensionRequest extends BaseRequest
{
    private String apiName;
    private Boolean updatePastEvents;
    private CustomDimensionModel customDimensionModel;

    public EditDimensionRequest(String apiName,
                                CustomDimensionModel customDimensionModel)
    {
        this.apiName = apiName;
        this.customDimensionModel = customDimensionModel;
    }

    @PathParam("apiName")
    public String getApiName() { return apiName; }
    public void setApiName(String apiName) { this.apiName = apiName; }

    @QueryParam("updatePastEvents")
    public Boolean getUpdatePastEvents() { return updatePastEvents; }
    public void setUpdatePastEvents(Boolean updatePastEvents) { this.updatePastEvents = updatePastEvents; }

    @BodyParam
    public CustomDimensionModel getCustomDimensionModel() { return customDimensionModel; }
    public void setCustomDimensionModel(CustomDimensionModel customDimensionModel) { this.customDimensionModel = customDimensionModel; }

    public EditDimensionRequest withUpdatePastEvents(Boolean updatePastEvents){
        setUpdatePastEvents(updatePastEvents); return this;
    }
}

There you have it: a very simple HTTP client built with Feign, using a request object. It made coding our functional tests way easier and made them much cleaner. The next steps for us would be to combine that with better exception handling (awesome post by my friend Jacques-Etienne Beaudet, go check it out) and we would be unstoppable!

Source: source.coveo.com

How to prevent frequent JavaScript mistakes

When writing JavaScript, I spend a lot of time fixing simple mistakes. Unlike
compiled languages you are more likely to make mistakes. It is easy for syntax
errors to sneak into your code without realizing it until you actually try and
run your code.

How many times have I got an undefined variable because I refactored some code
and forgot to rename that variable.

Even though it has been more than 5 years since I wrote my first Hello World.
The feeling remains the same – Why did I make this mistake again ?

To help me fix some of those mistakes, I tried a few linting tools over the
years. From the overly strict JSLint to the more
flexible variant JSHint and JSCS. I
recently discovered ESLint and fell in love with its
extensibility and features.

JSCS and ESLint have merged since April 14th you can check their blog posts JSCS and ESLint.

Overview

Linting is a process of checking the source code for programmatic as well as
stylistic errors. A Lint or a Linter is a program that supports linting.
They are available for most languages like CSS, Python, JavaScript, HTML, etc…

ESLint

ESLint is the most recent of the four linting tools
previously mentioned. It was created by Nicholas C. Zakas in june 2013.

Its goal is to provide a pluggable linting utility for JavaScript.

Designed to be heavily extensible, it comes with a large set of custom rules and
it is really easy to install. It gives precise and concise output by including
the rule name out of the box. You are always aware of which rule was causing an
error.

ESLint provides good documentation for its
rules. It is easy to follow and is grouped into
logical categories. Each rule gives details about what it enforces or not
and provides examples of good and bad written code for it.

Pros

• Customizable: every rule can be toggled, and many rules have extra settings
  that can be tweaked
• ES6/JSX support out of the box
• Supports custom reporters
• Has many plugins available and is very extensible
• Include many rules not available in other linters

Cons

• Slow, compared to JSHint or JSCS, but faster than these two if combined
• Requires some configuration to get started

Extensibility

Since ESLint is extremely extensible, I have created a shareable config rule
set to help my fellow colleagues
here at Coveo who write Pure JavaScript.

Open-sourcing

eslint-config-coveo started as an internal project for my team (Salesforce
integration) then we decided – Hey… why not open-source it?

With that in mind, I created a gulp task wrapper that uses our rule set
defined from that shareable
config. You can find the project
pretty-javascript here.

Want to be a JavaScript high priest of digital innovation?

For anyone who wants to write JavaScript like we do it at Coveo follow these
simple steps to get you up and running.

• Install pretty-javascript and gulp packages from npm

npm install --save-dev pretty-javascript gulp

• Create an eslint configuration file

.eslintrc.yaml

---
  extends: coveo
  rules:
    ... (they can be overriden)

OR

.eslintrc.json

{
  "extends": "coveo",
  "rules": {
    ... (they can be overriden)
  }
}

• Create a gulp task to run eslint

gulpfile.js

var gulp = require('gulp');
var linter = require('pretty-javascript');

gulp.task('lint', function() {
  gulp
    .src('src/**/*.js')
    .pipe(linter());
});

Want to be a TypeScript tech-wizard-in-residence?

For TypeScript lovers, Coveo also provides a gulp task wrapper to lint
TypeScript. The project pretty-typescript can be found
here (huge kudos to Dominique
Bégin).

Similarly to pretty-javascript, you only have to include pretty-typescript
and gulp from npm.

npm install --save-dev pretty-typescript gulp

Then simply add a task to lint your TypeScript code in your gulpfile as
follows :

var linter = require('pretty-typescript');
var gulp = require('gulp');

gulp.task('lint', function() {
  gulp
    .src('/src/**/*.ts')
    .pipe(linter())
    .pipe(gulp.dest('src'));
});

Sit back, relax and enjoy watching your silly mistakes from your terminal
output!

Source: source.coveo.com

Coveo JavaScript UI for Newcomers

The new Coveo JS UI has been available for a while now. How about getting off to the right start? Well you’re at the right place, here is a tutorial on how to configure and use the new Coveo JS UI.

Step # 1: Installing the Coveo Search API

The Coveo Platform 7 comes with the Coveo Search API, a web service offering a REST interface that is used by other Coveo products such as the Coveo JavaScript Search interfaces to send query and receive search results from a Coveo unified index.
The Coveo Search API REST endpoint can also be used by custom applications (see REST Search API Home).

As shown in the following diagram, the Coveo Search API acts as is a bridge between the front-end search interfaces or applications and a Coveo Enterprise Search (CES) instance maintaining a unified index on the back-end.
You can install the Coveo Search API on a server of your choice such as on the Coveo Master server (where Coveo Enterprise Search is installed), on a front-end server (where the Coveo JavaScript Search search is served from), or any other server.
Once installed, the Coveo Search API runs as a Windows service.

When your deployment includes one or more Coveo JavaScript Search interfaces, you must install and configure a Coveo Search API.

Follow the instructions described here in order to install the Coveo Search API. Then you’ll be asked to customize and start the Coveo Search API.

Here is an example of a working “config.yml” file:

Once you started the “Coveo Search API” service, you can validate that the service is accessible from various computers:

a.	Using a browser, access the URL in the following format:

"http://[REST_API_SERVER]:[port]/rest/search"
![Rest URL Example](/images/JSUI101/RestURLExample.png)

b.	Validate that the REST API JSON response appears in the browser
![Rest API JSON](/images/JSUI101/RestAPIJSON.png)

Step # 2: Creating the JS UI search page

Now that your Coveo Search API is well configured and up and running, you can now create your Coveo search interface in JavaScript.

+Using a browser, access the URL in the following format: “http://[REST_API_SERVER]:[port]”
+
+
+
+Click on “CLICK HERE TO START”
+
+
+
+Type your username and password, then click on connect, you’ll get the message below:
+
+
+
+On the “CREATE A SEARCH PAGE” window, you can click on “CREATE PAGE” if you just want to only use the “All Content” search interface, or you can click on “MORE TABS” in order to add more search interfaces.
+
+
+
+By clicking on “MORE TABS”, you’ll see the out of the box search interfaces available

+

+Click on the ones that you want to add into your search page. As an example, let’s click on “People”, “Email” and “SharePoint”

+

+Click on “CREATE PAGE”, you’ll get the message below:

+

+Here is your Coveo JavaScript search page:

+

+You are good to go! But do not stop there, there is so much more to do with it! Start here

Source: source.coveo.com

Sandboxing JavaScript Execution in Java

The Query Pipeline on a Coveo index can be extended using JavaScript code running on the server. This has many benefits, but allowing arbitrary code to run in a server process opens the possibility for an admin to bring the server down in many creative ways. To protect against this we added functionality to the JS interpreter we’re using (DynJS) to prevent Bad Things from happening.

There are three ways external code running in a process can cause harm:

• Using APIs to damage the server process or underlying server
• Using an excessive amount of CPU
• Allocating an excessive amount of memory

Blocking Access to APIs

At the core, JavaScript doesn’t provide any API that might be used for nefarious purposes. For example, there is no way to access the filesystem, or perform network requests. Still, JS interpreters in the Java world typically allow JS code to access any Java class present in the class path, and DynJS is no exception.

For example, JS code can use a Java object like this:

var foo = new java.util.ArrayList();

What we did is add a configuration option for DynJS to prevent it from exposing any Java package or object except the ones standard in JavaScript, plus any API that is explicitly exposed by the hosting application (which we assume are “safe”).

This shows how to set the option when creating a DynJS runtime object:

Config config = new Config(getClass.getClassLoader());
config.setSandbox(true);
DynJS dynJs = new DynJS(config);

Pretty simple, right? It was simple to implement too. Here’s how it works:

if (!runtime.getConfig().isSandbox()) {
    defineGlobalProperty("Packages", new JavaPackage(this, null), true );
    defineGlobalProperty("java",     new JavaPackage(this, "java"), true);
    defineGlobalProperty("javax",    new JavaPackage(this, "javax"), true);
    defineGlobalProperty("org",      new JavaPackage(this, "org"), true);
    defineGlobalProperty("com",      new JavaPackage(this, "com"), true);
    defineGlobalProperty("io",       new JavaPackage(this, "io"), true);

    defineGlobalProperty("System",   System.class, true);
}

We simply prevent any public symbol from being registered that would allow code to “escape” the sandbox. Without those, it can only reference stock JavaScript APIs.

Restricting CPU and Memory Usage

Calling JavaScript code from a server process is done in a blocking manner, which means that the call only returns when the code has finished executing. If that code decides to loop forever… well it helps with the heating bill, I guess.

To address that we need a way to stop script execution if it takes too long. But even that is not enough: the JS code might sometimes block on long-running network calls, for example, so strictly checking the time taken might block legitimate operations. What we want is a way to monitor the actual CPU time used.

Turns out this is pretty easy to do. The ThreadMXBean object exposed by the JVM provides methods to check the total amount of CPU time used by a thread. By reading the value just before script execution starts and then periodically checking the value during execution, it’s possible to detect when the script code has exceeded a pre-determined CPU quota. When this happens, an exception is thrown to inform the caller of the situation.

So, how do we arrange to periodically check the CPU quota? We need to have this check performed at some place where the JavaScript interpreter must pass no matter what kind of infinite loop it’s in. In this case I’ve chosen to do that in the interpret method of BlockStatement, which is essentially when any scope like a loop or condition or function body is entered. There we call checkResourceUsage from ExecutionContext, which will relay the call if resource quotas are being enforced.

Here is how we check that the CPU quota hasn’t been exceeded:

private void checkCpuTime() {
    long current = readCpuTime(threadId);
    long delta = current - lastCpuTime;
    lastCpuTime = current;

    if (cpuTime.addAndGet(-delta) < 0) {
        throw new ExceededResourcesException("CPU usage has exceeded the allowed quota");
    }
}

A similar technique is used to monitor heap allocations. The same ThreadMXBean object also exposes metrics about how many bytes were allocated by the current thread (including memory that is no longer referenced, but’s that OK). By checking this metric in the exact same way as for CPU, we can detect whenever the thread has exceeded the allowed memory quota and put an end to its processing.

NOTE: The call reporting memory allocation for a thread is not available on all JVMs, but works well enough on Hotspot and I expect others are implementing it as well. Trying to use memory quota on a JVM without the proper support would result in an UnsupportedOperationException.

To run JS code with quota checks, you need to run the code using the ExecutionContext returned by createResourceQuotaExecutionObject on an existing ExecutionContext. It is possible to invoke several blocks of code using the same quotas. The total consumed resources by all script executions will be used to check the quotas.

context.createResourceQuotaExecutionObject(new ResourceQuota(cpuQuota, memoryQuota));

What about Nashorn?

Nashorn is the new JS interpreter bundled with Java 8. It has good performance and is certainly more robust, but I still haven’t figured out a way to implement proper CPU and memory quotas in that engine, mainly because I haven’t yet found a place where I can regularly check if quotas have been exceeded (I haven’t tried very hard though). I might write a new post when/if I succeed in that endeavour.

Trying it out

The changes we’ve made to DynJS are available publicly on GitHub. We also submitted a pull request but it hasn’t been merged yet.

Update – February 2016

As of now, DynJS is no longer being maintained, and no official version has ever been released with the changes described in this post (in fact the PR is still opened).

Source: source.coveo.com

Creating a new JS UI component in TypeScript

Behind the scenes, the Coveo JS UI framework is built entirely in TypeScript. Obviously, it’s intended to be customized in JavaScript, but you may want to go further and create your own component in TypeScript. Be aware that this uses the internal JS UI APIs, so it can eventually break when updating. Purpose of the post is to give a glimpse of how the components are built internally.

Huge disclaimer : I am definitely not a TypeScript expert, and I am not working in the JS UI team. It is possible that my code is not optimized. This is more to give a basic guideline 😀

Huge disclaimer #2 : This article also implies that the reader has basic notions of TypeScript and the Coveo JS UI.

If you have ever downloaded the Coveo JS UI framework, you may have noticed that there’s a folder named lib in there. This folder contains the TypeScript definitions files we will need.

A component? What are we even talking about?

The JS UI is basically made of components, which are the different parts you can simply drop in your page. It goes from the facets, sorts, result lists to more advanced stuff such as the folding or analytics. On an architectural point of view, it is important to understand that a component should have a single responsability and should (at least try to) not impact others.

So, what do we want to build?

My use case was fairly simple : I wanted to create a component I called the ToggleResultList. This component would be a simple button allowing you to have different result lists in the same page (probably with a different style) and toggle between them. The main goal is to have something I can drop in my markup like this :

<span class="CoveoToggleResultList" data-result-list="#MainResultList"data-icon="default-result-list"></span>

Where the MainResultList is the ID of the HTML element containing the result list. Something like :

class=“CoveoResultList” data-wait-animation=“fade” id=“MainResultList” data-result-container-selector=“#ResultsContainer”> …

For details on the options on the Result List, you can refer to the Developers documentation on the component.

The TypeScript frame

So, let’s start by building the most basic component we can.

/// <reference path="../lib/CoveoJsSearch.d.ts" />

module Coveo.Test {
    export class ToggleResultList extends Coveo.Ui.Component {
        static ID = 'ToggleResultList';

        constructor(public element: HTMLElement,
                    public options?: any,
                    bindings? : Coveo.Ui.ComponentBindings) {
            super(element, ToggleResultList.ID, bindings);

        }
    }
}

Coveo.Ui.CoveoJQuery.registerAutoCreateComponent(Coveo.Test.ToggleResultList);

Let’s take a small breath, look out the window, and then focus on what we just wrote. For now, it doesn’t do a single thing, but the frame is there. We start by referencing the Coveo Js Search definition file, which will allow us to compile the whole thing. Then, we create our own class that extends Coveo.Ui.Component, which is the base class for any JS UI component. We then need an ID. This will get interpreted as CoveoToggleResultList in the markup, allowing anyone to drop this element in their page.

The constructor takes 3 parameters : the actual HTML element, any options that could be set on the component (we will come back to this further) and the current bindings (such as which search interface we are in). Don’t forget to call the basic constructor!

Finally, we use the framework to register the component. This line is really important, as it will indicate the JSUI to consider your block of code as an authentic component. From now on, you could compile your TypeScript code and integrate it in your page, right after CoveoJsSearch.min.js.

Be sure to check out Will’s excellent blog post on how to create a successful build process.

Adding some functionality

We have a component, it’s kinda cool! But it would been even cooler if it actually did something… Let’s add some stuff.

/// <reference path="../lib/CoveoJsSearch.d.ts" />

module Coveo.Test {
    export interface ToggleR1esultListOptions {
        resultList: HTMLElement;
    }

    export class ToggleResultList extends Coveo.Ui.Component {
        private static coveoResultListClass = '.CoveoResultList';
        private static disabledClass = 'coveo-disabled';

        static ID = 'ToggleResultList';
        static options: ToggleResultListOptions = {
            resultList: Coveo.Ui.ComponentOptions.buildSelectorOption({ defaultFunction: () => $(ToggleResultList.coveoResultListClass).get(0) })
        };

        constructor(public element: HTMLElement,
                    public options?: any,
                    bindings? : Coveo.Ui.ComponentBindings) {
            super(element, ToggleResultList.ID, bindings);

            this.options = Coveo.Ui.ComponentOptions.initComponentOptions(element, ToggleResultList, options);

            Assert.exists(this.options.resultList);

            this.bind.onRoot(Coveo.Events.QueryEvents.querySuccess, (e: JQueryEventObject, args: Coveo.Events.QuerySuccessEventArgs) => this.handleQuerySuccess(e, args));
            $(this.element).click(() => this.handleClick());
        }

        private getClassName(): string {
            return '.' + Coveo.Ui.Component.computeCssClassNameForType(ToggleResultList.ID);
        }

        private handleQuerySuccess(e: JQueryEventObject, data: Coveo.Events.QuerySuccessEventArgs) {
            if (!$(this.options.resultList).coveo().disabled &&
                !$(this.options.resultList).is(':visible')) {
                $(this.options.resultList).show();
            }
        }

        private handleClick() {
            $(this.getClassName()).addClass(ToggleResultList.disabledClass);
            $(this.element).removeClass(ToggleResultList.disabledClass);
            $(ToggleResultList.coveoResultListClass).coveo('disable');
            $(ToggleResultList.coveoResultListClass).hide();
            $(this.options.resultList).coveo('enable');
            $(this.getBindings().root).coveo('executeQuery');
        }
    }
}

Coveo.Ui.CoveoJQuery.registerAutoCreateComponent(Coveo.Test.ToggleResultList);

As you can see, we added some options in there. Those options are interpreted as data attributes in the markup. We will then be able to associate the component to a specific result list. You may wonder why we created a static options class…

1. It’s how the JS UI components are built.
2. It makes them compatible with the interface editor, which would allow anyone to simply drag and drop your component in a new or existing search page. (that’s for the interface part)

In our options for now, we simply added a ResultList, which is a selector option (meaning it points to another HTML element). There’s a huge variety of options you can have, simply use the autocomplete feature in your favorite IDE to see them 😀 What is really important is in the constructor, we need to initialize the options, this will make the link with the component.

We then bound our component to two events : the regular jQuery click event and the JS UI’s query success event. You can refer again to the Developers documentation to learn more about these events. The click event is responsible to disabling the other result lists and then to trigger a query on the one that should be shown.
==> It is important to actually disable and not just hide other result lists, since in the case the events would stay bound on them, so it would mess with your search page state.

You may also wonder why we show the result list in the querySuccess event instead of the click one… simple : we need to make sure the query has enough time to be executed. If we were to show it right away, it would “flick” a few milli-seconds and not be enjoyable for the user.

Adding mooaaaarrrr options

So, we have a component working, isn’t this nice? If you’re really building something that other developers might use, you may want to add even more options to your component to make it really awesome.

/// <reference path="../lib/CoveoJsSearch.d.ts" />

module Coveo.Test {
    export interface ToggleResultListOptions {
        defaultResultList?: boolean;
        icon?: string;
        numberOfResults?: number;
        resultList: HTMLElement;
    }

    export class ToggleResultList extends Coveo.Ui.Component {
        private static coveoResultListClass = '.CoveoResultList';
        private static disabledClass = 'coveo-disabled';

        static ID = 'ToggleResultList';
        static options: ToggleResultListOptions = {
            defaultResultList: Coveo.Ui.ComponentOptions.buildBooleanOption({ defaultValue: false }),
            icon: Coveo.Ui.ComponentOptions.buildIconOption(),
            numberOfResults: Coveo.Ui.ComponentOptions.buildNumberOption({ defaultValue: 10 }),
            resultList: Coveo.Ui.ComponentOptions.buildSelectorOption({ defaultFunction: () => $(ToggleResultList.coveoResultListClass).get(0) })
        };

        private iconTemplate = _.template("<span class='coveo-icon <%=icon%>'></span>");

        constructor(public element: HTMLElement,
                    public options?: any,
                    bindings? : Coveo.Ui.ComponentBindings) {
            super(element, ToggleResultList.ID, bindings);

            this.options = Coveo.Ui.ComponentOptions.initComponentOptions(element, ToggleResultList, options);

            Assert.exists(this.options.resultList);

            if (!this.options.defaultResultList) {
                $(this.options.resultList).coveo('disable');
                $(this.options.resultList).hide();
                $(this.element).addClass(ToggleResultList.disabledClass);
            }

            this.bind.onRoot(Coveo.Events.QueryEvents.buildingQuery, (e: JQueryEventObject, args: Coveo.Events.BuildingQueryEventArgs) => this.handleBuildingQuery(e, args));
            this.bind.onRoot(Coveo.Events.QueryEvents.querySuccess, (e: JQueryEventObject, args: Coveo.Events.QuerySuccessEventArgs) => this.handleQuerySuccess(e, args));
            $(this.element).click(() => this.handleClick());

            this.render();
        }

        private getClassName(): string {
            return '.' + Coveo.Ui.Component.computeCssClassNameForType(ToggleResultList.ID);
        }

        private handleBuildingQuery(e: JQueryEventObject, data: Coveo.Events.BuildingQueryEventArgs) {
            if (!$(this.options.resultList).coveo().disabled) {
                data.queryBuilder.numberOfResults = this.options.numberOfResults;
            }
        }

        private handleQuerySuccess(e: JQueryEventObject, data: Coveo.Events.QuerySuccessEventArgs) {
            if (!$(this.options.resultList).coveo().disabled &&
                !$(this.options.resultList).is(':visible')) {
                $(this.options.resultList).show();
            }
        }

        private handleClick() {
            $(this.getClassName()).addClass(ToggleResultList.disabledClass);
            $(this.element).removeClass(ToggleResultList.disabledClass);
            $(ToggleResultList.coveoResultListClass).coveo('disable');
            $(ToggleResultList.coveoResultListClass).hide();
            $(this.options.resultList).coveo('enable');
            $(this.getBindings().root).coveo('executeQuery');
        }

        private render() {
            var icon = this.options.icon;
            if (icon != "") {
                $(this.element).prepend(this.iconTemplate({icon: icon}));
            }
        }
    }
}

Coveo.Ui.CoveoJQuery.registerAutoCreateComponent(Coveo.Test.ToggleResultList);

You may notice there’s now 3 more options of different types.

1. You can specify if the component is targetting the “default” result list. This means this will be the one shown by default, and the other will be hidden at the beginning (in the previous example, you would have to do it manually).
2. You can specify the number of results of the result list. Normally, you would specify it on the whole Search Interface, but you may want to display a different number according to which result list is shown. We’re hooking on the buildingQuery event to change the number of results according to the options.
3. You can specify an icon! Using an UnderscoreJS template, this is a simple commodity to render nicely an icon for your component.

Wrap it up!

Now, my real use case was to give the user the possibility to toggle between a “regular” view and a “tabular” view. My markup looks like this, where the TableResultList and MainResultList are two different elements containing the two different result lists templates :

class=“coveo-toggle-result-list-section”> class=“CoveoToggleResultList” data-result-list=“#TableResultList” data-number-of-results=“50” data-icon=“table-result-list”> class=“CoveoToggleResultList” data-result-list=“#MainResultList” data-default-result-list=“true” data-icon=“default-result-list”>

If you wonder, it is located just under the search box in a typical JS UI search page.

And the visual result looks just like this:

Thanks a lot for reading! 😀

Source: source.coveo.com