Contribute

Last updated: jan.2, 2017 - 17:15:00

Overview

Hesperides consists of two parts :

  • a backend module developed in Java 8 (repo git hesperides")
  • a frontend module developed in Angular JS (repo git hesperides-gui)
Hesperides uses Redis to store values in list form. Each modification (of a module, a techno, platform (value properties)), it stores in JSON format. Here's a quick idea of architecture :

Development standards

Guidelines

If you have a patch or pull request for us, thank you! The more of these guidelines you can follow, the easier (and faster) it is for us to evaluate your code and incorporate it into the development branch. (Borrowing from ThinkUp's Pull Request Checklist):

  1. Make sure you can perform a full build and that the tests all pass. (See Building and Testing)
  2. Please try to add unit tests to demonstrate that your submission fixes an existing bug, or performs as intended.
  3. Rebase your branch on the current state of the master branch. Use git rebase master. This means you might have to deal with conflicts.
  4. Consolidate commits so that all related file changes are within a single commit. You can squash commits into a single commit by doing git rebase -i.
  5. Be descriptive in your commit messages: explain the purpose of all changes. You can modify commit messages by doing git commit --amend (for the previous change), or git rebase -i for earlier changes.
Once your branch is cleaned up, please submit the Pull request against the master branch. Thanks for your help! A typical workflow:
# get up to date with origin/master and create a branch
git checkout master
git pull
git checkout -b issue/#(issueNumber)
# now make your commits.
$EDITOR somefile
git commit -am "[issue#(issueNumber)] - A good description of what you did"
# If you are now out of date with master, update your master:
git checkout master && git pull
# now go back to your branch and rebase on top of master
git checkout - # "-" will checkout previous branch
git rebase master
# resolve any conflicts, and possibly do `git rebase --continue` to finish rebasing
# now push your branch to your fork, and submit a pull request
git push myfork newbranch

Documentation

Remember to update the documentation to help cronies who come after you !

Backend

Follow the traditional development standards.

Tests are mandatory

Tests for backend are mandatory. Otherwise your application will be refused.

To execute all the JUnit tests:

mvn test

To execute a single one:

mvn -Dtest=HesperidesPropertiesModelTest#shouldCreateModelFromCode test

Frontend

Follow the traditional development standards.

Styles

For frontend, do not use style = "..." in HTML files : styles are set in css files.

Webservices on Hesperides

As you already know, Hesperides backend exposes webservices that are requestes by frontend or directly by other external tools. And you know also that services are based on dropwizard. Let's see now how to add a new webservice in Hesperides !

Processing unit

You have to start with the implementation of your webservice. To do that you have to create :

  • a subpackage in com.vsct.dt.hesperides
  • a java interface containing the signature of your methods
  • an implementation of your interface : that will be your "Aggregator"

Tests

Consider your tests now ! (a contribution without tests will not be merged)

Your webservice

Once your processing unit is ok (with unit tests), you can now expose it in REST API Hesperides. To do that you have to :

  • Create the resource class of your service in the following package : com.vsct.dt.hesperides.resources. This class must have the following form :
    • it has to contain an instance of your aggregator as attribute
    • it has to contain a constructor that takes all parameters
  • Write your webservice

Make it available

Even if your webservice exists, it's still not available for hesperides. To make it available, you have to add an entrypoint in dropwizard : the run method of MainApplication

  1. Inform dropwizard to handle with your aggregator. To do that, you have to create an instance of your aggregator and make it manageable. Example :
    /* Events aggregate */
    EventsAggregate eventsAggregate = new EventsAggregate(eventBus, eventStore);
    environment.lifecycle().manage(eventsAggregate);
  2. Inform dropwizard of the existence of your web resource. Example :
    // Events resource
    HesperidesEventResource eventResource = new HesperidesEventResource(eventsAggregate);
    environment.jersey().register(eventResource);

Finished ?

Not really ! you just have to restart your bakend !

Services architecture

Hesperides backend is made of follwing services :

  • application
  • file
  • module
  • template
  • version
A class Hesperides<service>Resource matches each service (for example HesperidesApplicationResource).

An aggregate is also linked to Hesperides<service>Resource (example : ApplicationAggregate). This aggregate :

  • handles with the business rules and cache of the service
  • is also used by other aggregates to receive events

Aggregates have a parent class SingleThreadAggregate. This class allows only one thread at same time : this mecanism avoids concurrent modifications (for example the modification of the same application).

Identified change

This limit has been identified. There may be a future development on this subject.

Some classes are registered in event bus to allow indexation of modules, templates and applications in ElasticSearch :

  • ModuleEventsIndexation
  • PlatformEventsIndexation
  • TemplateEventsIndexation

This allows to isolate indexation and aggregates. In fact, as you can see in the diagram above, when an aggregate is executed, it transmits message on event bus, that will be pick up by the other aggregates and the event indexation.

Events

Hesperides doesn't store structured object, but instead store every steps (event) that allowed to build this objet.

Example

127.0.0.1:6379> keys module-demo*
1) "module-demoKatana-war-1.0.0.0-wc"
2) "module-demoKatana-war-1.0.0.1-wc"
127.0.0.1:6379> llen module-demoKatana-war-1.0.0.1-wc
(integer) 4
127.0.0.1:6379> lrange module-demoKatana-war-1.0.0.1-wc 0 999
1) "{\"eventType\":\"com.vsct.dt.hesperides.templating.modules.ModuleCreatedEven
t\",\"data\":\"{\\\"moduleCreated\\\":{\\\"name\\\":\\\"demoKatana-war\\\",\\\"v
ersion\\\":\\\"1.0.0.1\\\",\\\"working_copy\\\":true,\\\"technos\\\":[],\\\"vers
ion_id\\\":1},\\\"templates\\\":[{\\\"name\\\":\\\"application.properties\\\",\\
\"namespace\\\":\\\"modules#demoKatana-war#1.0.0.1#WORKINGCOPY\\\",\\\"filename\
\\":\\\"application.properties\\\",\\\"location\\\":\\\"/appl/{{username}}/prope
rties\\\",\\\"content\\\":\\\"spring.datasource.jndi-name=jdbc/demoKatana\\\\nin
fo.propriete_en_dur={{propriete_a_surcharger}}\\\\n{{#test}}\\\\n\\\\t{{prop2|co
mmentaire}}\\\\n{{/test}}\\\",\\\"version_id\\\":1}]}\",\"timestamp\":1448975559
026,\"user\":\"my_user\"}"
2) "{\"eventType\":\"com.vsct.dt.hesperides.templating.modules.ModuleTemplateUpd
atedEvent\",\"data\":\"{\\\"moduleName\\\":\\\"demoKatana-war\\\",\\\"moduleVers
ion\\\":\\\"1.0.0.1\\\",\\\"updated\\\":{\\\"name\\\":\\\"application.properties
\\\",\\\"namespace\\\":\\\"modules#demoKatana-war#1.0.0.1#WORKINGCOPY\\\",\\\"fi
lename\\\":\\\"application.properties\\\",\\\"location\\\":\\\"/appl/{{username}
}/properties\\\",\\\"content\\\":\\\"spring.datasource.jndi-name=jdbc/demoKatana
\\\\ninfo.propriete_en_dur={{propriete_a_surcharger|@required}}\\\\n{{#test}}\\\
\n\\\\t{{prop2|commentaire}}\\\\n{{/test}}\\\\n{{trucu}}\\\",\\\"version_id\\\":
2}}\",\"timestamp\":1449049683051,\"user\":\"my_user\"}"
If we have a look at this example:
  • the first record is a com.vsct.dt.hesperides.templating.modules.ModuleCreatedEvent event : this means that at this time, "my_user" created a module "demoKatana-war" with a template "application.properties" etc ...
  • the second record means that the user "my_user" updated template "application.properties"

Conclusion :

To reconstitute the entire module, you must play all records.

Version id

A field is really important : version_id.

This field increments each time an update is made in an object (that can be a template/module/techno/platform). This attribute ensures data consistency. Use case :

  • When hesperides-gui gets a template, it gets version_id
  • If you updates this template and saves it, hesperides-gui returns version_id
  • Hesperides backend check version_id matches the last version : if not, the change is rejected

Types of events

For platforms/applications :

  • PlatformCreatedEvent : creating a platform
  • PlatformCreatedFromExistingEvent : creating a platform from an existing one
  • PlatformUpdatedEvent : updating a platform (for example : update of the version)
  • PlatformDeletedEvent : deleting a platform (deleting ElasticSearch index by the database Redis)
  • PropertiesSavedEvent : updating valuation of your platform

For modules :

  • ModuleCreatedEvent : creating a module
  • ModuleUpdatedEvent : updating a module
  • ModuleWorkingCopyUpdatedEvent : creating a workingcopy module from another one
  • ModuleTemplateCreatedEvent : creating a template (file) in a module
  • ModuleTemplateUpdatedEvent : updating a template (file) in a module
  • ModuleTemplateDeletedEvent : deleting a template (file) in a module

For technos :

  • TemplateCreatedEvent : creating a template (file) in a techno
  • TemplateUpdatedEvent : updating a template (file) in a techno
  • TemplateDeletedEvent : deleting a template (file) in a techno
  • TemplatePackageDeletedEvent : deleting a techno

Object and ObjectData

In service inputs there are Objects, and in Redis we store the same kind of Object but suffixed by "Data".

The desire is to have immutable objects in the database. When a platform is read in database, a PlatformData object is returned.

This is the same object as the input of the service /applications/{application_name}/ platforms. The advantage is that the Platform object is immediately converted into PlatformData preventing modification (it will recreate an object for this).

Another advantage to build a PlatformData object manually (which does not come automatically from the database), you have to use a builder (constructor is private!). Thus, functional checks (mandatory data ...) are in the constructor.

This allows more robust and more maintainable code.

Model and Valuation

Overview

Hesperides works with a principle of Model and Valuation. On the one hand we have a model (property name, annotation ) and on the other side valuation (property name/value). Model and valuation are not store together in database.

Actually :

  • model is not store, it is deducted from template/module. In fact, with the module (and its associated templates), we can extract the model : that's what Hesperides does.
  • platform (application) stores properties keys and values.

Example

If we consider the module module-test, version 1.0 and template appliaction.properties with the following content :

spring.datasource.jndi-name=jdbc/demoKatana
overload_property={{overload_property|@comment "some other comments" @required}}
password={{password|@comment "some comments" @password}}
default={{default|@default foo}}
regular_expression={{regular_expression|@pattern t.*t}}

Calling REST service /modules/module-test/1.0/workingcopy/model, we get this :

{
  "key_value_properties": [
    {
      "name": "regular_expression",
      "comment": null,
      "required": false,
      "defaultValue": "",
      "pattern": "t.*t",
      "password": false
    },
    {
      "name": "default",
      "comment": null,
      "required": false,
      "defaultValue": "foo",
      "pattern": "",
      "password": false
    },
    {
      "name": "password",
      "comment": "some comments",
      "required": false,
      "defaultValue": "",
      "pattern": "",
      "password": true
    },
    {
      "name": "overload_property",
      "comment": "some other comments",
      "required": true,
      "defaultValue": "",
      "pattern": "",
      "password": false
    }
  ],
  "iterable_properties": []
}

When REST service is called, the class HesperidesModuleResource receive it. Then we get the module (with class ModuleAggregate) and for each file we ask to mustache to give properties (class TemplateSlurper).

Now to get properties values, as they are stored in the platform, we can call REST service /applications/MYAPPLICATION/platforms/MY-PLATFORM/properties?path=#FIRSTLEVEL#SECONDLEVEL#module-test#1.0#WORKINGCOPY :

{
  "key_value_properties": [
    {
      "name": "overload_property",
      "value": "FOO"
    },
    {
      "name": "regular_expression",
      "value": "tfgehzrgyierhgergert"
    },
    {
      "name": "password",
      "value": "{{hesperides.application.version}}"
    },
    {
      "name": "missing",
      "value": "Where am I ?"
    }
  ],
  "iterable_properties": []
}

With model and platform, we can make the connection :

  • the "default" property is not valuated because it is in the model but not in the platform
  • the "missing" property has been deleted in the template because the property is in the platform but not in the model

Cache and snapshot system

Implementation

Hesperides use Guava cache.

Caches are implemented in their class :

  • ModuleRegistry,
  • PlatformRegistry,
  • TemplatePackageRegistry.

Cache creation are done by HesperidesCacheBuilder class. HesperidesCacheBuilder class allow to get a standard configuration in YAML Hesperides file.

For each cache, you can specify a size and a TTL

cacheConfiguration:
  redisConfiguration:
    type: REDIS
    host: 127.0.0.1
    port: 6379
    retry: 5
    waitBeforeRetryMs: 20000
 nbEventBeforePersiste: 5
 templatePackage:
    maxSize: 100
    #weight: -1
  module:
    maxSize: 100
    #weight: not valid
  platform:
    maxSize: 100
    #weight: not valid
  platformTimeline:
    maxSize: 100
    itemExpireAfter: 10m
    #weight: not valid

Not supported

Platform cache and Module cache do not support weight parameter because they are a simple objet and not a list.

To their cache, a loader is associate :

  • TemplatePackageCacheLoader,
  • PropertiesTimelineCacheLoader,
  • PropertiesCacheLoader,
  • ModuleCacheLoader.

Their loader class load item. To do that, their use a event reconstructor :

  • VirtualApplicationsAggregate (+ VirtualPlatformRegistry),
  • VirtualModulesAggregate (+ VirtualModuleRegistry and VirtualTemplateRegistry),
  • VirtualTemplatePackagesAggregate (+ VirtualTemplateRegistry).

Virtual* class simulate non virtual class (same name class without virtual) to replay all events.

Virtual* class don't write in database !

This is the class diagram :

*Registry class have guava cache and use *CacheLoader to load missing item in cache.

*CacheLoader read events from database and replay it.

Their events are cached by Virtual* class to rebuild item. When done, item store in Guava cache.

Snapshot system

Because, to get final state of item (Hesperides must be replay all events) a snapshot is take regularly to increase boot start time.

When we create a platform, we create a associate snapshot (with 1 event).

Then, when we update platform, a new snapshot is create if more than X event (nbEventBeforePersiste in configuration file) are store since last snaphost.

Whe a platform already exists, Hesperides check if platform is in memory cache (Guava). If not, Hesperides load last snapshot and replay just last event that stored after snapshot.

If no snasphot found in database (when new system is actived), Hesperides replay all events from scratch.

Release version of template package and module have no snapshot cause, their contain only one event.

Auto regeneration system :

If number of event in database is less than the last snapshot. Hesperides drop snapshot and regenerate all.

Indexation

You can run a REST service to reindex ELS :

  • URL : http://IP:PORT/rest/indexation/perform_reindex
  • METHOD : POST

Important Notice

Regardless the kind of request, they should all be done against the back-end of hesperides. The back-end will query against Elasticsearch himself