Pour les candidatures, veuillez fournir un CV à contact@inovia.fr

En tant que Chef de Projet Big Data, vous assistez le Directeur de Pôle sur la coordination et le développement de projets à forte volumétrie. A travers ce poste, vous pourrez acquérir une expérience riche dans une société de référence du web français ; vous développerez vos compétences de gestion de projet orienté data.

DESCRIPTIF DU POSTE

• Animer une équipe de 2 développeurs et 1 data analyst.
• Rédiger les spécifications fonctionnelles et techniques détaillées.
• Former son équipe aux best practices Inovia.
• Réaliser les chiffrages et les plannings de l’équipe.
• Contribuer au savoir faire du pôle de 7 personnes.
• Reporter au directeur de pôle le suivi projet.
• Formaliser les tâches d’études pour les ingénieurs.
• Monitorer la balance Qualité / Risque / Prix / Délai du projet.

La mission envisagée se concentre sur un ensemble d’études de données analytics à forte volumétrie : de 1 million de lignes à 60 milliards.

PROFIL

• Diplôme d’ingénieur Grandes Ecoles, spécialisation IT.
• Anglais courant indispensable.
• Expérience personnelle ou professionnelle en développement web et en gestion de projet.
• Maîtrise parfaite du développement PHP et du SQL. Notions de MDX appréciées.
• Des compétences en Hadoop sont un plus.
• Vous êtes curieux des innovations technologiques.
• Esprit d’analyse et d’organisation, soucieux du détail.
• Plaisir d’apprendre et d’exceller.
• Culture générale importante.

MODALITES

• Titre de transport pris en charge à 100%.
• Rémunération à 39k brut minimum (selon expérience).
• Formation continue aux méthodes de gestion CMMI.
• Séminaire annuel au ski d’une semaine entièrement pris en charge.
• Sur Paris 9e, dans des locaux de qualité supérieure.
• Chaine de production rôdée et très bien outillée.
• Equipe de haute volée, reconnue internationalement.

Nous vous fournissons tous les atouts de la réussite. Souhaitez vous avoir l’expérience et le plan de carrière que vous espérez depuis toujours ?

Pour les candidatures, veuillez fournir un CV à contact@inovia.fr

Pour les candidatures, veuillez fournir un CV à contact@inovia.fr

En tant que Chef de Projet, vous assistez le Directeur de Pôle sur la coordination et le développement de projets à forte composante Mobile et Backoffice. A travers ce poste, vous pourrez acquérir une expérience riche dans une société de référence du web français ; vous développerez vos compétences de gestion de projet web.

DESCRIPTIF DU POSTE

• Rédiger les spécifications fonctionnelles et techniques détaillées

• Animer une équipe de 4 personnes

• Former son équipe aux best practices Inovia

• Réaliser les chiffrages et les plannings de l’équipe

• Contribuer au savoir faire du pôle de 8 personnes

• Reporter au directeur de pôle le suivi projet

• Formaliser les tâches de développement pour les ingénieurs

• Monitorer la balance Qualité / Risque / Prix / Délai du projet

La mission envisagée se concentre sur la réalisation d’une plateforme à haute disponibilité, forte criticité et très interconnectée. La plateforme est web et mobile.

PROFIL

• Diplôme d’ingénieur Grandes Ecoles, spécialisation IT

• Anglais courant indispensable

• Vous disposez d’une expérience personnelle ou professionnelle en développement web et en gestion de projet

• Vous maîtrisez le développement PHP et le JavaScript

• Vous êtes curieux des innovations technologiques

• Esprit d’analyse et d’organisation, soucieux du détail

• Plaisir d’apprendre et d’exceller
• Culture générale importante

MODALITES

• Titre de transport pris en charge à 100%

• Rémunération à 38k brut minimum (selon expérience)

• Formation continue aux méthodes de gestion CMMI

• Séminaire annuel au ski d’une semaine entièrement pris en charge

• Sur Paris, 9e, dans des locaux de qualité supérieure

• Chaine de production rôdée et très bien outillée

• Equipe de haute volée, composée de 20 experts

Nous vous fournissons tous les atouts de la réussite. Souhaitez vous avoir l’expérience et le plan de carrière que vous espérez depuis toujours ?

Pour les candidatures, veuillez fournir un CV à contact@inovia.fr

Free web tech testing and nice job offers

You want to test you knowledge about web technologies? Then come and pass the test! A few questions will be asked. The best of you will maybe get one of the 5 positions currently opened @Inovia. Good money, good environment in one of the most skilled team in Paris!

Check out the past Zend Framework position described on our website to see the advantages we have to offer.

Discussions and debates

Just come to discuss with us about your vision of cloud computing, big data or x-commerce. We will have a few drinks, candies and we will demo you some of our products and projects.

Here is the list of the different conferences held:

• 14.00 : Travel shopping by Amadeus
• 15.15 : Smartphone development by Aubey
• 16.30 : BI 2.0, new challenges by Soft Computing

Here is a map to access the site: Map for Horizons Informatiques

Sounds great? So see you @Horizons Informatiques!

During the recruitment phase, you’ll get to pass two job interviews: one by phone, the other one in our office. During the first interview on the phone, we’ll ask you some technical questions to evaluate your technical level.
So get a job interview and evaluate yourself for free!

Job description

This job is the perfect experience for young talented developers.
Your responsibility will be first to work with the rest of the team on our open source libraries and integrate them inside our clients applications.
Then the position can progress to operational management (3 developers) within a year.

General qualifications

• min. master or engineering degree
• immediately operational in PHP5 / SQL / HTML / CSS
• Good theoretical knowledge
• fluent in English and French

Required Technical Skill Set

• PHP 5 OO
• Design patterns and software architecture
• HTML / CSS / JS
• PgSQL / MySQL
• Eclipse

Major plus bonus

• Zend Framework
• ExtJS, JQuery
• CI, Unit testing
• Magento

Context

The team is welcoming and skilled, the office is spacious. You will have a lot of freedom and support in your daily job.
We keep track of state-of-the-art technologies and like innovative approaches.
Position opened in Paris, France.
Revenue: 37.5k€ minimum.

Adding structure is adding value

Without structure, the most visited attraction would have been only a bunch of metal.

As we saw earlier, the more the data are structured the more potential your data have. It is only a matter of hiding the complexity to each group of users, which is not always simple. So let’s stop thinking about the data and take a look at the application and the users.

If we need to evaluate the value of a BI app, one approach I often use is R.E.C.: Reference, Empowerment, Comfort. To empower users, we must have a data model as close as possible as people see their job. The more meaningful, relevant structures we can provide, the more value the app will have. So this is definitely a functional question.

So now that we know we need to provide the most meaningful data structures to users to maximize the value out of our application, these questions remain:

• when data are unstructured at the origin, what can we do to structure them, in a meaningful way?
• Should I use a db or a search engine for this?

The example: a list of recipes

A search engine is way better to process natural language than a database. It is also really good to fasten full text search. In general, search engines are great to extract info from unstructured structures. But should the data warehouse be also unstructured for BI purposes?

We will discuss where the value of data is located, and see if the functionalities of current search engines are sufficient. A concrete illustration of partially unstructured data is a list of recipes in full text. Let’s say you have this info about a recipe :

Recipes can be really unstructured ! courtesy of Pirate Johnny (c)

title : Burger

course category : entree
complexity to prepare: 1 out of 5
time to prepare: 10 min
description: “ a delicious burger home made with a juicy steak, toasted bread, cryspy lettuce. Can be served with ketchup and love.”
Instructions : “Toast the bread first. Then cook in a frying pan the meat. Salt at the end of the cooking.”
source: “allrecipes.com”

Extracting business related terms: the thesaurus

From a data perspective, one can see a recipe as a collection, an assemblage of ingredients. A “burger” is composed of “bread”, “steak”, “cheese”, “lettuce” and “ketchup” for instance.
The simplest way to model this situation is to use a tag mechanism. So we use a Full Text Indexer to process the description and extract the tags. Of course, I’ll need to get a base of what is a ingredient and what is not (in our burger “love” is not an ingredient, even if some french people would say so). The list of recognizable words is called a thesaurus.
With the help of our Full Text Indexer and the reference thesaurus, each recipe now have a list of ingredient tags associated with it. Now I’m able to count the ten most used ingredients, count the most used ingredient with steak…
But is using a Full Text Indexer with a simple thesaurus sufficient?

Adding structure to the maelström of tags

Common situation is a synonym or a connex term : a “steak” is indeed a “beef steak”. When one wants to build indicators for all the beef recipes, we need to count a recipe with steak as a recipe containing beef. The tags are connected between each others and we must model this in our thesaurus.

What will really sublime the data is indeed the organization of the tags. This organization requires hierarchies (steak is a specialization of beef, which is a specialization of meat), segmentation (alcohol-free, alcool), collections (japanelo, tacos, burritos are in mexican family).
These structures can’t be handled right now by any search engine I know of, but on the other hand only a few database engines are able to manage a thesaurus imho. So here, a search engine combined with a database really make sense. The search engine for the Full Text Indexer capability, and the database for the structure aspect.

 
We saw in the first part how to dynamically create the needed partitions.
Just add this function in Postgresql (via psql for instance) :

 CREATE OR REPLACE FUNCTION run_on_partitions(text,text) RETURNS INTEGER AS $$
DECLARE
partition RECORD;
tablename TEXT = $1;
sql TEXT = $2;
sqlReplaced TEXT;
BEGIN
tablename := tablename || '%p';
FOR partition IN SELECT relname::text as rel FROM pg_class WHERE relname::text LIKE tablename AND relkind = 't' ORDER BY relname LOOP
sqlReplaced := replace(sql, '',partition.rel);
RAISE NOTICE 'Executing: %', sqlReplaced;
EXECUTE  sqlReplaced;
END LOOP;
RETURN 1;
END;
$$ LANGUAGE plpgsql;

 
This function is really useful. To call it, simply do:

 SELECT run_on_partitions('tablename','CREATE INDEX _idx ON  USING btree(name)');

 
The tag <PARTITION> will be replaced by the name of the partition derived from the master table ‘tablename’. We can now create indexes, primary keys on each table…without having to run the query against each table by hand.
Let’s create a full text search index on these items:

SELECT run_on_partitions('tablename','CREATE INDEX _fts_idx ON  USING gin (to_tsvector(name::text || '' ''::text) || description));');

 
The search time has now been greatly improved.
 
Yeepa!

It is going on for a week or so and I can tell you as a manager that this is great because:

• you earn time, don’t have to go outside to choose where you gonna eat, what and to end up queuing.
• Once a week, you will have to spend 45 min shopping goods and cooking for everybody, but cooking is definitely a more enjoyable activity than waiting to get your order. Inovia people like creative activity, and cooking is considered like this in France.
• you eat healthy food. So you are in a better shape for a great afternoon of work.
• you earn money, because cooking for 15 people or so cost definitely less than buying already made food. Count 3.5 euros per person, all included.
• you share a meal with your coworkers. This is the coolest part !

Luckily our office in Paris got an amazing kitchen where any food delight can come true.

We wish you a happy summer time !

Approach

Let’s say you want to partition a table based on time period.
In Postgres, we handle partitioning with inheritance. You will have a master table, used to regroup the different partitions, and one table per partition. I let you read the excellent Postgres documentation about this: http://www.postgresql.org/docs/current/static/ddl-partitioning.html

As you can see, there is a lot of tedious work to create partitions, tables, indexes… Here we provide you a non optimal solution in terms of insert performance, but definitely more practical in terms of risks and time to settle than the method presented in the manual.

I will show you how to conduct it in a simple case and simplify a bit the manual operation you need to do. From this, you’ll be able to build your own solution.
We got a large fact table called tablename. There is a column on this table called period which can have values like 2011_01 which means January of 2011.
We will create one partition per period. So we will create the tables:
- tablename_2010_01p
- tablename_2010_02p
- tablename_2010_03p
…

The p at the end helps us detecting that this object is a partition. I admit there is a large space to improve it.

Create the master table

-- Create the master table tablename_with_partition:
-- You should use the same exact schema as the one from the non partitioned table.
CREATE TABLE tablename_with_partition
(
id integer NOT NULL DEFAULT nextval('tablename'::regclass),
"name" character varying(150) NOT NULL,
description text NOT NULL,
period character varying(10) NOT NULL
CONSTRAINT tablename_with_partition_pkey PRIMARY KEY (id)
) WITH ( OIDS=FALSE);

Auto partition creation during insert

-- Attach a magic function to the insert of this table:
CREATE OR REPLACE FUNCTION create_partition_and_insert()
RETURNS trigger AS
$BODY$
DECLARE
partition VARCHAR(25);
BEGIN
partition := TG_RELNAME || '_' || NEW.period || ‘p’;
IF NOT EXISTS(SELECT relname FROM pg_class WHERE relname=partition) THEN
RAISE NOTICE 'A partition has been created %',partition;
EXECUTE 'CREATE TABLE ' || partition || ' (check (period = ''' || NEW.period || ''')) INHERITS (' || TG_RELNAME || ');';
END IF;
EXECUTE 'INSERT INTO ' || partition || ' SELECT(' || TG_RELNAME || ' ' || quote_literal(NEW) || ').*;';
RETURN NULL;
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;

This function will try to insert in the correct partition and creates it if it doesn’t exist. This approach is not the most efficient, but if you got less than 20M records, it works definitely fine.
Here we attach the function defined earlier to the insert operation on tablename_with_partitions:

CREATE TRIGGER tablename_insert_trigger
BEFORE INSERT ON tablename_wtih_partition
FOR EACH ROW EXECUTE PROCEDURE create_partition_and_insert();

Notice that I didn’t named the trigger with the name “tablename_with_partitions_insert_trigger”. It is because I planned to substitute the partitionned master table and the old non partionned version later on.

Migrate the data

-- Copy data from the non partitioned to the partitioned version:
INSERT INTO tablename_with_partition SELECT * tablename;

It can take a while. But it is a nice approach to do this way.

Testing everything is fine

Let’s check everything is ok:
- “SELECT count(*) FROM tablename_2010_1;”: you got proper data in one partition.
- “SELECT count(*) FROM tablename_with_partition;” returns the sum of all the elements in the partition
- “SELECT count(*) FROM ONLY tablename_with_partition;” returns 0. It is normal we didn’t insert one line of data in the master table.

What’s next ?

If everything is fine, we are almost done. We’ll need to build the indexes and substitute tablename by tablename_with_partition.
We’ll see this in the next article by creating a smart function that can execute on every partition an arbitrary SQL command.

Business intelligence is the science of gathering and structuring data to help making decisions. We will call Business Intelligence BI for short. One major technical challenge is that BI generally involves dealing with large volume of data, which cause troubles to the classic database approach. The search engines (SEs) are a new approach to handling data. Is a search engine a go-to solution for BI usage ?

SEs are particularly good for dealing with large amount of data. This quality is based upon smart architectural solutions to common problems. In essence a SE :

• crawls raw data
• indexes the different data (classify and interpret)
• provides functionalities to look for data related to a theme (search)

To illustrate our talk, we’ll take as an example a BI program that target employee’s emails. All the company’s communications are classified and then one can search for emails and computes indicators on this data.

Data structured vs. unstructured: how ambiguous your data are

The classification process in BI have to understand the meaning of each email. If I write down this information:

baptiste.manson@inovia.fr pierre.cornic@inovia.fr I’m not here now.

We can interpret different facts:

• Baptiste is not here and notified Pierre;
• Baptiste is not here and notified Pierre just now;
• Baptiste and Pierre are not here.

We can say that the data is ambiguous.

Now if the data we received are:

baptiste.manson@inovia.fr to pierre.cornic@inovia.fr said “I’m not here right now”.

Now we understand without ambiguity what is the related fact to the piece of information:

• Baptiste said to Pierre that he wasn’t here when he sent the email.

It is due to the new structure elements that we added: baptiste.manson@inovia.fr *to* pierre.cornic@inovia.fr *said* *”*I’m not here right now*”*.

To be able to understand what is going on in the company, we need the data to be structured.

Search engines are particularly known for dealing with unstructured data. They are performant when a user wants to compare an unstructured question to unstructured documents, like books or websites. Take the most famous one: Google. Finding an exact match can be complicated.
For instance, I looked for what obama said to sarkozy last week and I typed : “obama said to sarkozy last week”… I find first a quote from Sarkozy to Obama. Then the next links are related to common declarations of the two men together. Only the fourth link was about a quote from Obama to Sarkozy.

The factual details described in my search and in the websites are not correctly interpreted.
Another issue is that “last week” is badly intepreted, as I wanted to look for declarations within last week, and not articles containing the words “last week”. The root of the issue is the unstructured aspect of the data I was looking for.

The business item concept

We saw that structured data are essential in a BI context to remove ambiguities. SEs are not particularly designed for dealing with structured data, even though a solution exists: one should deal with “business items”. This concept, proposed by Exalead, is in practice really hard to maintain. Check this article for further info. We’ll see why in the next paragraph.

Consistency of the data: what is conform

My first paragraph was about data structure. The new SE support data structure to some extend, introducing the concept of “business item”. This concept brings clever solutions to performance wise technical issues but also introduce functional limitations.

The first obvious limitation is to ensure consistency. Consistency is to make sure that every single data respects a standard. For instance, that an email has a sender, and this sender must exist in our user referential. The conformity checks must be supported at all cost by the application using the SE. As there is no centralized description of the whole structure with constraints, it implies that every developer must check carefully the details. To be fair, large database based applications in BI must also drop some of the conformity check features for performance reasons.

Manage changes: how the data repository can live after its conception

The second problem is that It is common in BI to go back on already stored data and discover a new way to use them, a new interesting indicator to build. Like splitting the mail addresses in two, one part for the name and another for the domain.

Usually, data inside a SE are dead data, they cannot be changed in a new way. On the contrary, a database can easily update later data already stored.
Sure, Solr and Exalead provide such functionality but it is in a limited upsert way (cancel and replace). It means that if one wants to update data, it needs to extract every business items, change it and push again in the repository. It is like not having the “search and replace” function in excel. This limitation causes headaches to engineers when the functional requirements are changing.
This can explain why some people that run a Proof of Concept around a SE as a main data repository can be disappointed later on because they can’t easily manage changes.

Se or not SE ?

One quality in SE is its speed to index large volume of unstructured data. But its defunct is to store data in an ambiguous way and not let users change the structure later on.

This is why at Inovia we decided to let the SE work only as an extension of a database. In our guidelines, the SE shouldn’t be the main repository of data.

We have a database that allows us to finely structure data and a search engine for the unstructured data when the database is not relevant anymore. Read this article for further info about the features of SE over databases.

Postgres got a mechanism like this called Full Text Search. We will discuss in another post a concrete business case that successfully mixed the two approaches, database and search engine.