Icelandic Open Data, lambs and maps
Two weeks ago I attended the first Open Knowledge Foundation Iceland hackathon. The main goal of the day was to work in some visualizations of already published information and, although the number of attendees was quite small as it’s usually the case with first editions, a few interesting ideas were proposed.
Between them, one that caught my imagination was bringing to life the data published by the The Icelandic Agricultural Advisory Centre about the quality of the lamb population in Iceland. This public institution provides professional consulting for farmers and every year they inspect 90% of lambs in iceland and publish a few reports with the results. Thanks to this work they can for example pinpoint the best ~300 farms in Iceland according to the quality of their lambs.
As unfortunately and usually happens with Open Data the first step to tell the story behind these reports is to convert them in machine-readable information. We know this is usually a point hard to understand by non-developers and institutions: the problem with publishing information in PDF is that is not possible to feed these documents into any kind of software in a reliable way. These are formats designed to be read by humans, not by computers. The input for algorithms ideally should an homogeneous, ornament-free, text only format. So that was our first step.
With the help of pdftotext, a utility for converting PDF files to plain text files, we did a first explorative conversion of the PDFs. As you can see the translation of a PDF like this into its text version is far from perfect but the layout is conserved on the crucial tabular data so it’s a good start. After some clean up and manual editing we got our first csv readable files:
Many people –and almost every Icelander– describe Icelandic lamb as the best lamb in the world. This is usually attributed to the fact that the animals roam free during the summer only to be collected again in September during a special meeting called Réttir where Icelandic sheep farmers invite family and friends to help out with the round-up:
Given the lambs stay in wild nature for many months subject to the weather and local climate, we thought it would be interesting to map its quality and some other features and see if we could see some geographical patterns on the data. The first problem we found was that the farm information was not geolocated.
Luckily, Iceland is a country that dedicates many resources to the management of geographical information, in part because the rift associated with the Mid-Atlantic Ridge, runs across the country from the southwest to the northeast. This rift marks the division between the European and North American tectonic plates, and moves at the not negligible spreading rate of ~2 cm per year giving Iceland its earthquakes, its volcanic nature and creating constant changes in the landscape.
The Icelandic National Registry provides access to geo information of every parcel in Iceland and although for the first tests I used some python code to query their server, we thought it would useful to extract and polish this piece of code into its own gem: skra-geo. With this working it was possible to correlate the name of a farm like Bjarteyjarsandur with its coordinates [378995, 435348] and update the csv files.
But wait a moment, those coordinates don’t look like the usual lat-lon pair… Yes, Iceland uses its own reference system called ISN93 (and its newest revision ISN2004). You can learn more about the reasons for its use in this presentation but the main reason is that the aforementioned tension in the rift deforms the network of reference GPS stations constantly, creating a context with very unique challenges. Again, we have open source’d a ruby gem called isnet93 that allows simple conversion between this system and the internationally supported WGS84.
Finally we were able to assign geographical coordinates to each farm in the reports and our csv files were ready to be put on a map. For this task we sticked to CartoDB a great mapping service that makes easy to build visualizations on top of tabulated data. After importing the data, we played with the configuration of the different visual styles and create a custom basemap. After more tweaking here you have an initial release of the map according to the fat in the lambs and the general quality:
Top icelandic farms in 2013 according to lamb
Two interesting details are evident in the these maps:
Although the quality is very high in all the country, it’s possible to detect small clusters of farms in areas like Grímsnes where it’s possible to find the best of the best. The reason for this concentration could be connected with different reasons like good local practices in the treatment of the sheep or specially good grazing areas.
In the second map, built using the amount of fat in the lambs of the best farms, we can see clearly a front running in the SW-NE axis in the isthmus of the Vestfirðir peninsula. This could initially correlated with the harsh winter conditions in this area, that forces the animals to be better insulated, but further research to discard other factors would be interesting.
We’ve had a lot of fun creating these maps and we’ve only started to scratch the surface of the information. For example, the RML has been publishing these reports for more than a decade now, and it’ll be amazing to integrate all the data to see trends over time. Modern tech tools and fast development iterations are bringing new possibilities to every industry… sheep farming included.
Conferences: be clear & respect your speakers
Last week my proposal “Scripting your way to (im)mortality” for Codebits was accepted. When I submitted the abstract, I knew the topic was not the usual, worn introduction-to-tdd talk and that, given the audience at this conference is composed by very talented technical people, getting selected would be difficult. Surprisingly, I got very positive feedback and the organizers announced its selection. They contacted me almost immediately after the tweet asking for any special requests/questions I could have. I asked for a few details (expected projector’s resolution, etc.) including what personal information they needed for coordinating my trip.
Last monday, one of the organizers contacted me to let me know that “there might be some confusion” since they “usually” don’t pay for any travel or accommodation expenses. Apparently “the only exception would be when we invite a high profile speaker who otherwise wouldn’t attend” and that in that case their budget “is very tight and usually amounts to about 5 people (speakers)”.
I could not believe what I was reading. Covering –at least– the travel and accommodation expenses is a common practice for all kind of events (including free ones). So if, for whatever reason, that is not the case they should state it very clearly in every speaker-oriented communication, including the Call For Papers. But the fact they give undisclosed preferential treatment to a number of speakers is just shocking. If you check the speakers lineup of most tech conferences, you’ll see an appalling number of repeated names. These are the “high profile” guys. This is the fuckingly boring conference circuit at work. Adding insult to injury, I know through personal contacts that the only-high-profiles expenses policy was not applied in previous editions.
When I started talking in spanish conferences many years ago I decided two things: I wouldn’t charge for presenting them and I wouldn’t pay –directly or indirectly–to give them. Let me explain the reasoning behind those two maxims.
Not getting paid for your talks
I believe in sharing ideas and knowledge because the simple process of presenting and discussing your thoughts in public have the marvelous collateral effect of making them clearer and more useful.
First, because nobody wants to be publicly exposed as a total idiot, and therefore all of us make big efforts to write down, edit, polish and present our ideas in the most intelligible way. Second –and more importantly– because once you’ve exposed your ideas, they are subject to exploration by others and that connected research is, in the end, what brings society good ideas. This mutualism between speaker and attendees is, at least for me, enough payment for delivering my talk.
Actually, I think this exchange is better achieved in unconferences, where the structure enabling the dialog between speaker and attendee –that is, the organization– is minimum. The usual bias in the organizers that creates the circuit is removed, and as consequence they are usually more diverse and interesting. That is reason why we sponsor events like Startup Open Space Zaragoza 2014.
Not paying for giving your talks
Actually by “not paying” I mean not paying more, because any –serious – speaker is already committing a big number of hours (aka money) to research, prepare, rehearse and deliver the talk without getting any economic compensation. As mentioned before I feel this time is well spent, but many conferences tend to forget that most speakers are running their talks in a nonprofit manner.
In this regard, I think the main role of the organizers should be to enable the most interesting speakers to share their ideas without taxing them with an extra load of expenses on top of the time and effort they are already devoting to their talk. For community and “free” events this means finding sponsors and partners to pay for the cost of creating a sharing context and –optionally– giving them some visibility. In for-profit events that means… well… basically using some of your damn profits to cover the cost of the people that make your event possible.
No matter what is the type of conference you are organizing you should respect the effort and time spent by your speakers to make your conference better and try to remove any other possible extra burden. If for some reason you can’t do so, it’s just fair to be clear from the very beginning about it.
Consuming your own APIs
As mentioned in our last weeknote, we’ve resumed the development of Stage’s API(s). I’ve been personally interested in an approach that seems to be slowingly morphing into a de facto standard in the IT industry: the creation and use of your own APIs as data source for your products. For a long time we’ve seen how different companies I deeply respect for their technological achievements have started using this technique:
Twitter has been dogfooding its APIs to build the twitter.com website for a few years now.
Heroku dashboard internally uses their ever increasing APIs to render user’s apps status pages and resource management tools.
Kickstarter uses an internal, not publicly announced API to feed its iPhone app. I found this extremely interesting since it denies the usual API-as-service-for-others point of view and focus on giving yourself a controlled test field for exploring what you want to enable external parties to do with your platform.
Although is hard to find public documentation on the building of these kind of systems, we’ve been collecting small bits and pieces here and there and last week we started writing some code to implement a integral API(s) for Stage. We already had a micro-core with basically read-only info about events and bookings but the idea is to grow this into a comprehensive API(s), complete enough to be used as main engine for our own product.
In my humble opinion, there are a few clear advantages to this approach:
Language independence. I think this is too often ignored: once you have an API acting as central provider of your service/product you have gained the ability to create consumers for it in any desired language. Linking Paths has been a Ruby-only company for years and we’ve been very happy with it, but it’s useful and reassuring to know that we can use different technologies when they are a better fit in a given context. APIs makes bootstrapping these situations much easier.
Increased device flexibility. As a collateral effect of the previous point, the building of device/context specific applications for your project can assume that the basic core of its functionality is done from day 1, since they’ll be using the API(s) as foundation.
Easier implementation of platform-wide policies. From monitoring to quotas, the API(s) become a clear coding point for all kinds of horizontal strategies with the confidence that you can impose then uniformly or to manage them in a case-by-case fashion (twitter’s own apps preferential treatment comes to my mind).
Raised awareness on your system reliability. When the downtime or bugs in your APIs affects only external, third-party systems is easy to get used to it or to simply underestimate the importance of these errors. Using the same channel –even if different conditions are applied to internal clients– increases the trust that all the parts have on your service.
Promotion of distributed systems. An API(s) gives your service of a massive accessibility. It’s no accident that the documentation of many APIs shows how to access them with cURL from the command line. Thanks to this, the idea of creating small services that can reuse functionality is promoted, moving a product from the traditional monolithic approach to one that composed by many micro-services.
All those undeniable merits are not the only consequences of an API based product. There are, of course, some disadvantages:
Increased difficulty to change contracts. From the moment your API is made public and clients start using it you should be very careful changing this contract. The obvious countermeasure to this problem is to version your API, but strategies like using it privately for a long test period —like in the Kickstarter case mentioned before— are also worthy of being explored.
More complexity. Any new indirection layer adds complexity to a system and using an API to access your data is not different. In the same way, using an ORM, a MVC paradigm, or a specific web framework injects problems that simply don’t exist if you use SQL or write everything by yourself in a single file.
The common point between all this decisions however is that we –the tech community– realized many years ago that in most contexts these tools offered more benefits than problems and, therefore, made sense to use them.
It took me many years to realize that coding –and I’d dare to say life– is basically that: the art of managing trade-offs.
Very busy but not terribly productive week (worst combination ever). Working in different areas at the same time with limited resources often decrease immediate outcome in each one of them although in some cases –eg. when the purpose of the work is research/exploration– it’s the only way. At the same it’s infinitely more funny than sticking to a piece of code for 5 days.
This has been the case with the work done this week in Stage. We’ve started the development of its API(s) with the clear short-term goal of growing the micro-core we already had to the point we can use it internally as datasource for our own web app. We expect this to collaterally provide us with a robust API that will become eventually a core value for our customers. Next week we’ll publish a post about the logic behind this decision and the advantages/disadvantages it involves.
The work done with Verkami this week is noteworthy too. We’ve been working in the creation of a proper category system for the platform. So far we have been using a tag-based system for characterizing each project. These tags are fed into a system that enable Verkami’s staff to create dynamic selections of projects (eg. Halloween: projects tagged with horror and movie or book or music). This has been working pretty well for years –and will be maintained in the future– but we have detected too that a proper ontology where each project is related to one category will be useful to improve the presentation, discovery and statistical analysis.
Additionally a new partners program will be announced soon. It has been a long planned feature that we expect will promote discovery using your favorite magazines, institutions and websites to recommend you the most interesting projects.
Going back to Linking Paths, we’ve been thinking about the visual foundation of our company. I’ve felt many times in the past that there has been a lack of building blocks enabling a consistent and quick visual bootstrapping of ideas/products/sideprojects. A few months ago I read about base, a styleguide and CSS framework by the smart guys at Mapbox, a nice implementation of this idea. As part of a potential redesign of our website we’re getting more and more excited with the idea of having a live reference of our visual identity. Lots of work ahead.
Finally two possible side projects have received a few hours of attention this week: first —unnamed still— is a small iOS app to discover food’s seasonality. It’s too soon for talking about it but the idea is connected with the exploration of API-first services and one of my obsessions, gastronomy.
Second project —unnamed too— is a variation of a previous idea to provide in-depth, realtime analytics to physical locations. In a first approximation to the problem I got stuck due to the cost of the infrastructure to setup for each new location. A few days ago, reflecting on the almost universal use of smartphones in some countries, brainstorming for creating a super cheap setup started. Both side-projects are right now just written ramblings on notebooks but who knows… maybe they will see the light of day soon!
So, that’s it! Have a great week!
January is flying by! This week has been devoted to plan 2014 with customers. Big new features on the roadmaps, ideas to strengthen their teams and products and lots of energy to make this a great year.
One of the things I’ve committed myself this year is to led Linking Paths with two main values: happiness and meaningfulness. With this two states on mind I’ll pursue a great year not only for me, but for my customers, my and their projects and eventually anyone that gets in contact with the company.
One of the first efforts to materialize this view was expected to happen this weekend. The project –codenamed 'Bila', basque for search– has been our first humble attempt to transform year kickoff meetings into inspirational experiences. As mentioned at the beginning of the post, it’s normal practice for many companies to schedule special meetings at the beginning of every year with the intention of define the goals and objectives for the new year. These meetings are a good tool to organize the work, but the fact that they are usually attended by the same people, with the same agenda and in the same space makes them a horrible tool for inspiration.
I wanted to change all this by transforming the usual year-planning private meeting into a semi-public inspirational event. The idea: invite one of Linking Paths customers to spend a weekend exchanging ideas with a multidisciplinary (designers, marketing, makers, etc.) team of selected brilliant people interested in their core business.
To increase the intensity of the exchange we’d host the event and accommodate all the attendees in a nice countryside house in the Basque Country. To make it more refreshing –and interesting for everyone– the schedule will be filled up with secret activities visiting companies from different sectors –but again connected to our customer’s core business– that I think embody the idea of innovation.
Opening the event to people that is not in daily contact with your product or service brings fresh air to the conversation and forces you to be open to new ideas. Additionally short visits to companies doing interesting stuff is, at least for me, one of the best energy boosters. Add a group of smart folks to the mix and you’ve the perfect combination for a happy beginning of year at work.
Unfortunately I’ve been forced to delay the event in the last moment due to some unexpected cancellations on the attendees group. Although a lot of work was made preparing the event, I’ve decided to postpone it before seeing the idea implemented in a poor way. I’m sure it’s a great way to make Linking Paths relationship with our customers more meaningful and it’ll be done sooner of later. No need to rush.
So that was our week. I hope you all have had a great beginning of year. Let’s see what else does 2014 bring us. Rock&roll!
Addendum: chavalina mentioned in the comments that an event like this can be pretty hard to achieve.
I’d like to clarify that the event was in fact 99% ready: activities booked, external businesses/companies willing to participate, accommodation rented, attendees and customer ready to rock… I’m pretty confident the experience will be great if we can find the right moment to do it.
The most difficult part of doing this –and the reason why there is not a lot of companies doing it– is that in order to make it low-risk for the customer and attractive for the attendees, we were going to pay for everything and celebrate the event during a weekend. “Only” thing we were asking them was two days of their free time. And if you look at it from a purely financial point of view it doesn’t make sense for Linking Paths. More costs and no economic profit for a meeting that we could do in the traditional way.
But that is exactly what I was trying to highlight: the benefit of this kind of initiatives is not economical but relational. We care about our customer’s businesses to the point of investing our time and money on them, inverting the traditional approach to a client-partner interaction. We are confident that this will make our relationship more solid and trustable and that is, on itself, the best asset.
Week #458 – Reboot
It was 1999. My second year as developer/project manager in my first job in the software industry. I remember the moment clearly.
I’ve been on the phone for about 15 minutes solving a problem with a customer. The issue is finally solved and I’m hanging the phone. The administrative officer of the company enters the developers’ room, where I’m the only person at the moment to make a photocopy.
She’s a nice forty-something woman that sees and hears everything in the office: books plane tickets, arranges meetings, pays and generates invoices, etc. She fills a strange space in the company’s hierarchy, in permanent contact with the CEO but considered a coworker by every employee.
Waiting in front of the copier, she stares at me for a few seconds and, without any introduction, says:
"You’ll do big things… but not here."
The tone is friendly and positive, and oracle-esque omen of good times ahead. Then she gets the paper and leaves the room, never to mention the conversation again.
For some reason I don’t fully understand, the cryptic moment got stuck in my mind. During years it grew inside me, with the force and bitterness of the truths that one don’t have the courage to admit. A deep sense of unfulfilled potential has always been harbored in my heart.
I joined Linking Paths in 2007 and being cofounder of the company brought me very good moments. However, the period 2009-2012 was the hardest, most demanding and stressful period of my professional life. It took a heavy toll on my mood and my health. During this period I made big sacrifices on many areas of my personal life that now I deeply regret. Thankfully, during all these bad moments I’ve had the unconditional support of my wife. I’d never been able to carry on without her help.
Last April I became principal and only owner of Linking Paths. I kept working on our ongoing projects and products but intentionally stopped writing in the blog and looking for new projects. As planned, I’ve used 2013 to think about what is a company, why I run a company, what is the goal of Linking Paths and pondering my own reactions to those questions. I don’t have clear answers for all these questions, but I’ve many conclusions and ideas on them that I’ll like to share with you in future posts.
Fifteen years later I’m rebooting myself, and in the process I’m rebooting my company. I invite you to join me in the process.
On April 1st, 2005 I founded Linking Paths. By that time I didn’t know almost anything about running a company, especially after the main idea behind it fell apart seven days later. And I’m still not sure about what is this all about, but that’s another story. I’ve an announcement to do today and felt this birthday a round date for it.
This is something that some of you, dear readers, already knew. A few of you guessed it since I haven’t write any weeknote for the past months. But today is finally becoming formal, public and final: yesterday was my last day at Linking Paths.
But this is not its end. Aitor has taken the lead and I’m sure he will do many memorable things in the coming time. That’s is what he has always done since he joined me in 2007. I made this decision for personal reasons, I really feel it’s time for a change. For me and for Linking Paths.
I would like to use this last post to thank you all of you, friends, customers and occasional readers, for your support all this time. It has been a long and worthwhile journey and I really appreciate it. I’m very proud of what we’ve achieved together in this eight years, even with its not-so-funny moments. Thank you very much for being there.
That’s all. Take care.
PS: To satisfy the curious. I’m taking a couple of months off while closing the transition for some customers that took longer than expected. No plan after that yet.
Another week finishing… and more work done in multiple fronts. This week was rough in Iceland due to weather conditions but as with the postal carriers neither snow nor rain nor heat nor gloom of night stays us from our deploys.
Support and some billing in Stage. The week started on Sunday’s night with some issues related to our fees. As we state clearly in our product’s homepage our fee for free tickets is 0. The reason for this policy is that we’ve always supported people organizing open, community-oriented, free events providing them our service for free. However many organizers promote their events giving discounts and coupons to customers that sometimes cover the full ticket price and we think that is purely a marketing decision, a good one probably but marketing nonetheless and therefore the 0 fee doesn’t apply.
Alberto met with Bazaar’s founders during the week to push the project further and continue development in the first shop powered by the system. The shop is designed to interact with a central catalog system (Bazaar) and one of the main challenges is to ensure a good performance and stability for this interaction.
In the last weeks some testing on Foundation 4 was done. Although the new version has been a complete revamp and there are obviously some small bugs to be ironed out in the coming weeks, experience has been quite solid so far. It’s well structured, configurable and mobile-first. It’s a very interesting tool for us: you can prototype an app in no-time, there is a good number of add-ons and the documentation is clear and simple.
Facebook integration in Verkami is basically done. The internationalization process is madness and Open Graph custom actions must be manually approved by Facebook, but now everything is in place for slowly rolling out the feature to users. Most of the implementation is transparent to users and not terribly complex but I’m happy with it because is configurable (letting users to share more/less info in their timelines), extendable (allowing verkami staff to add new sharing options in the future) user-oriented (detecting and connecting new facebook users with their old verkami users, conserving contribution history) and multi-provider (enabling other OAuth platforms -eg. Twitter- to be used as extra identity in the future).
The recommendation for the weekend is a little wink to all the nerds in the audience that -like us- spent many hours in the 90’s playing with a very special type of cards: Friday Night Magic.
Have a nice weekend!
Resist discouragement. Your own and the one you acquired from others. Resist the sleepless nights and the after-hours work, especially the one that doesn’t make sense. Hold that unnerving feeling when something inside your brain says what the hell am I doing. Grit your teeth and move on, specially when it would be easy to turn around and do something else. Assume that things will not be right the first time, probably not even in a second try but trust that, still, hard work and do-the-right-thing are almighty tools.
Accept that the encouragement of your family, wife and friends is a gift too precious to allow yourself to fail when facing adversity. Resist the thought of failure, the though of “shit happens”. Be true to yourself and don’t let yourself be impressed by the pyrotechnic flaunting of fops and supernovae. Hold on to the dream, the enthusiasm, the primal desire that started everything in first place, beyond all logic reason and beyond all doubt. Persevere and work hard to honor the people who support you.
When things go wrong, rectify and keep fighting. This will put you in the same group in which the greatest humans in history are; no one is remembered by never been wrong. Build something you can be proud of with trust, honesty, transparency and dialogue. Resist the lie of work-life dichotomy: improving our work improves our lives and 8 hours per day is a lot to put them in a corner at night. Prove that profitable and human are not incompatible terms.
Resist. Resist. Resist.
If you would like to know how to make a company work you just have to learn what that word means.
I wrote this five years ago.
Have a great weekend.
413 Request Entity Too Large: The request is larger than the server is willing or able to process.
The HTTP is wise and universal: that is the perfect definition for this week. Is easy to fall into despair when you try to cope with many projects and fail miserably doing it… but one must move forward. A few things done this week (very similar to what has been done during February):
- Integration of the front-end in the Bazaar’s shop project.
- The usual support and maintenance in Stage.
- Estimates and inquiries for multiple project propositions.
- The first private alpha integration of Facebook OAuth & Open Graph in verkami has been deployed.
To research this last point, we’ve tested many well known sites doing similar stuff and discovered that most implementations are really poor. One small example: most apps don’t try to correlate new identities coming from Facebook with existing users in their sites (with the consequent confusion and loss of historical information for the users). Additionally this precludes users from disconnect their facebook accounts in the future if they change their mind. We’re working hard to build a better approach.
My recommendation for this weekend is to cook. Yes, you’ve read correctly: cooking. I could write pages and pages about all the links I can see between cooking and programming but that is probably material for a special post and not the weeknote. If you want some nice inspiration be sure to check this three examples of great gastronomical tips and recipes:
They are mainly a collection of short, super-high production quality videos made by a small team of enthusiastic foodies. Very inspirational.
Enjoy the weekend!