Playing with MongoDB

MongoDB is a nosql database. This means that you don’t use SQL to query the data, instead you use javascript and receive responses in json objects. I enrolled in course M001: Mongodb basics to get a basic understanding of how it works and it seems extremely easy.

mongodb-for-giant-ideas-bbab5c3cf8

The cool part about it being javascript based is that you can become a “full stack” web developer just by learning javascript. The front end with angular, and the backend with node.js and mongodb can be sufficient for you to build web apps. (I still shudder thinking that javascript is being used for anything, yet here we are..)

Mongodb is extremely easy to start playing with, for free you can create a database cluster in the cloud with 512MB by registering in mongodb.com and getting a Mongodb atlas.

MongoDB Compass is a graphical client with which you can connect to your database and analyse your data. It analyses the data and gives you insights on the types of data, you can also filter just by selecting with your mouse the data you want to filter on in an intuitive way.

The CLI is the best way to power-query your data. below my cheat-sheet after completing the basic training course:

MongoDB cheatsheet

  1. Show databases
    1. Shows the databases
  2. Show collections
    1. Once within a database , shows the collections
  3. Insert data
    1. db.moviesScratch.insertMany([],{“ordered”:false)
      1. With ordered:false, it does not stop when a duplicate _id key is found,
  4. Query data
    1. db.movies.find().pretty()
  5. Comparison operators
    1. Greater than
      1. db.movieDetails.find({runtime:{$gt:90}})
    2. Greater than and Less than
      1. db.movieDetails.find({runtime:{$gt:90, $lt:120}},{_id:0,title:1,runtime:1})
    3. Greater or equal, less or equal
      1. db.movieDetails.find({runtime:{$gte:90, $lte:120}},{_id:0,title:1,runtime:1})
    4. Not equals
      1. db.movieDetails.find({rated: {$ne:”UNRATED”}},{_id:0,title:1,runtime:1})
    5. In
      1. db.movieDetails.find({rated: {$in: [“G”,”PG”]}},{_id:0,title:1,runtime:1,rated:1})
  6. Filter for null or non existing fields
    1. db.movies.find({mpaaRating:{$exists:true}}).pretty()
    2. db.data.find({atmosphericPressureChange:{$exists:false}}).pretty().count()
  7. Filter fields by type (double, int, string…)
    1. db.movies.find({viewerRating:{$type:”double”}})
  8. Filter with multiple “OR”
    1. db.movieDetails.find({$or: [{“tomato.meter”:{$gt:95}},{“metacritic”:{$gt:88}}]},{_id:0,title:1,”tomato.meter”:1,”metacritic”:1})
    2. db.shipwrecks.find({$or:[{watlev:”always dry”},{depth:0}]}).count()
  9. Filter with multiple “AND” (this is only needed when we want to apply many filters on the same key
    1. db.movieDetails.find({$and: [{“metacritic”:{$ne:null}},{“metacritic”:{$exists:true}}]},{_id:0,title:1,”tomato.meter”:1,”metacritic”:1})
  10. Array field moderators
    1. Get all objects which contain elements in array
      1. db.movieDetails.find({genres:{$all: [“Comedy”,”Crime”,”Drama”]}},{_id:0, title:1, genres:1}).pretty()
    2. Get size of an array
      1. db.movieDetails.find({countries:{$size:1}}).count()
      2.  db.data.find({sections:{$size:2}}).count()
    3. Find a single object with multiple elements which match all the criteria
      1. db.movieDetails.find({boxOffice: {$elemMatch: {“country”: “Germany”, “revenue”: {$gt: 16}}}})
      2.  db.surveys.find({results:{$elemMatch:{“product”:”abc”,”score”:7}}}).count()
  11. Regular expressions
    1. db.movieDetails.find({“awards.text”: {$regex: /^Won.* /}}, {_id: 0, title: 1, “awards.text”: 1}).pretty()
Advertisements

Docker cheat sheet

Another year, another euskalencounter.org 🙂 This year I’ve binge watched Altered Carbon, and katakoda’s Docker training, below is my cheat sheet for future reference.

  1. docker search java
    1. search for a docker image (java in this case)
  2. running a docker image
    1. docker run -d openjdk
      1. run detached (-d) the openjdk docker
    2. docker run -d –name redisHostPort -p 6379:6379 redis:latest
      1. –name creates a user friendly name for your running docker
      2. -p <machine host-port:container-port> exposes docker image port in the host port
      3. redis:latest is the docker image:the version
    3. docker run -d –name redisDynamic -p 6379 redis:latest
      1. -p 6379 assigns a random host port
    4. docker port redisDynamic 6379
      1. shows to which host ip:port is docker running image redisDynamic’s port 6379 is assigned
    5. docker run -d –name redisMapped -v /opt/docker/data/redis:/data redis
      1. -v host-path:docker-path maps a docker path to a host path in order to persist data
    6. docker run -it ubuntu bash
      1. -it runs an interactive docker image.
      2. you can run a specific command on the docker image (bash)
    7. docker run -d –name my-production-running-app -e NODE_ENV=production -p 3000:3000 my-nodejs-app
      1. -e VARIABLE=value sets an environment variable in the running container
  3. How to run a command inside a running container
    1. docker exec -d ubuntu_bash touch /tmp/execWorks
  4. How to create docker data volumes
    1. Create a new container to start it afterwards (instead of running it directly) this is useful mainly to create data storage dockers and use them from other images.
    2. docker create -v /data –name dataContainer ubuntu
    3. docker run –rm –volumes-from dataContainer ubuntu ls -la /data
    4. docker cp config.conf dataContainer:/config/
      1. copies file config.com from the host to the docker container
    5. to move data containers from one machine to another you can export/import them to .tar
      1. docker export dataContainer > dataContainer.tar
      2. docker import dataContainer.tar
      3. docker import http://example.com/exampleimage.tgz
  5. How to create a network between containers
    1. docker network create backend-network
      1. creates a network to which we can attach containers. Docker sets a dns server in ip 127.0.0.11 and all connected containers use it by docker adding it to resolv.conf on each container
    2. docker network connect frontend-network redis
      1. connects a container to a network
    3. docker network connect –alias db frontend-network2 redis
      1. –alias <alias_name> creates an alias for the network name
    4. docker network ls
      1. shows all networks
    5. docker network inspect frontend-network
      1. shows all containers connected to a network and their ip addresses
    6. docker network disconnect frontend-network redis
      1. disconnect a container from a network
    7. docker run -d –name=redis –net=backend-network redis
      1. launches a redis container linked to the network backend-network.
  6. Building a docker image
    1. docker build -t webserver-image:v1 .
      1. -t <image name:tag> is used to give a name and version to the image.
      2. in the . directory you need to create a file called Dockerfile with yaml notation
    2. .dockerignore
      1. echo passwords.txt >> .dockerignore
        1. all fine names included into file .dockerignore will be ignored when building the image
    3. Dockerfile
      1. FROM nginx:alpine
        1. uses nginx:alpine as base for this docker image
      2. COPY . /usr/share/nginx/html
        1. copies from the host to the image
      3. FROM node:7-alpine
      4. RUN mkdir -p /src/app
        1. RUN’s command inside the docker image
      5. WORKDIR /src/app
        1. set the working directory in the docker image
      6. EXPOSE 3000
        1. EXPOSEs the port from the docker image to a random port in the host system
      7. CMD [ “npm”, “start” ]
        1. runs a command in the docker image you can add parameters as items of the array
      8. ONBUILD COPY . /usr/src/app
        1. onbuild commands won’t be executed until the built image is used as a base image
    4. docker images
      1. returns a list of images installed in the host system
  7. removing a docker image
    1. docker container ls -a
      1. lists all containers (running or not)
    2. docker container rm < CONTAINER ID>
      1. removes the container
    3. docker rmi hello-world
      1. removes image hello-world
  8. docker ps
    1. show docker processes
  9. docker inspect openjdk
    1. show details of running docker
  10. docker logs openjdk
    1. shows logs for running docker

100+ C-Level insights from Moneyconf

This month I returned to moneyconf, a yearly meeting of fintech enthusiasts. Apart from a pair of trendy LHoFT sunglasses I came back with a bunch of insights and inspiration to fuel me until next year.

moneyconflhoft

Insights from moneyconf:

  1. 20 million SME in the USA don’t accept electronic payments today @thefriley @Square CFO
  2. The finance part of fintech hasn’t been disrupted yet (as opposed to retail) there’s a lot of legacy just signing an nda can take 3months. @thefriley @Square CFO
  3. “When we work with the banks it is slow going…” @thefriley @Square CFO
  4. The internet deserves a native currency (bitcoin) cashapp allows to seamlessly buy/sell bitcoin. Fiat has a lot of friction, a lot of value lost in sending money overseas. This shouldn’t be the case. I’m all in to disrupt that. @thefriley @Square CFO
  5. I believe the way to get great decisions is to have diversity. If everyone agrees, someone is not thinking. Inclusion is needed for diversity to stay. Feminine role models in leadership inspire. Only 5% women in leadership, there is systemic bias @thefriley @Square CFO
  6. If you only interview women you will be able to find a woman (surely in the whole world there will be one woman for the job) push your hr team to find them. @thefriley @Square CFO
  7. The internet is broken. It has no digital identity construct. With blockchain we can have identity. Social context can attest they are a citizen. Zug in switzerland is an experiment on cryptovoting @ethereumJoseph Consensys Founder
  8. Blockchain is a force for universal disintermediation. Intermediaries will be right-sized as to how much value they extract from a transaction. @ethereumJoseph Consensys Founder
  9. com disintermediation for music. Civil, fake news control. Data storage, heavy conputation, kwh… all is being disintermediated @ethereumJoseph Consensys Founder
  10. Web 3.0 will bring native digital money, greater security, trusted transactions, decentralised storage, bandwidth, heavy compute… @ethereumJoseph Consensys Founder
  11. Bullshit sharing economy (uber, lyft, airbnb…) vs. actual sharing economy will happen thanks to ethereum smart contracts with networked business models @ethereumJoseph Consensys Founder
  12. Siloed infrastructure is moving towards collaborative, distributed infrastructures. Enterprise ethereum alliance has over 500 members building this shared infrastructure @ethereumJoseph Consensys Founder
  13. We can imagine a dystopian world where we have no privacy. We can imagine a world where privacy is everywhere. Regulators have a big role on this. @JoelKatz Chief Cryptographer at Ripple
  14. Xrapid is a ripple product which exchanges usd for xrp. Xrp is sent to an exchange that takes mexican pesos. These xrp are then changed for pesos. This is done in under two minutes in a way which is cheaper and faster with current remmitance systems @JoelKatz Chief Cryptographer at Ripple
  15. Volatility if xrp is lower in two minutes than the mexican peso over two days. @JoelKatz Chief Cryptographer at Ripple
  16. The real utility of blockchain hasn’t happened yet. This is the reason for the big volatility. @JoelKatz Chief Cryptographer at Ripple
  17. In 20 years payments will be invisible, inexpensive, and seamlesss. Similar to opening a webpage today @JoelKatz Chief Cryptographer at Ripple
  18. Utility tokens are an aberration. If you are raising capital it’s a security. I’m surprised it took so long for the SEC to step in. Patrick Byrne @OverstockCEO
  19. ICOs enable financial inclusion. Access to capital for any enterpreneur in a third world country. This is groundbreaking. @jeffpulver
  20. It’ll be possible for the underbanked to receive loans peer to peer without a bank Patrick Byrne @OverstockCEO
  21. Blockchain will bring more chain than the internet did. It’s a bigger disruption than the internet was Patrick Byrne @OverstockCEO
  22. Formation of capital, central banking, cap markets, identity, supply chains,voting will be disrupted by blockchain. We will be able to give a collapsing economy a central bank as a service. Patrick Byrne @OverstockCEO
  23. Regulators have good understanding of the blockchain revolution. It’s too late to stop it. In some countries the regulators work for the companies they should regulate. Patrick Byrne @OverstockCEO
  24. Bitcoin’s been through 6 booms and busts. If you zoom out it doesn’t look like a bubble. There’s more and more widespread, infrastructure is growing. @DavidFBailey CEO at BTC Media
  25. Bitcoin offers decentralised trust. Censorship resistant. There is an intrinsic value in this. @DavidFBailey CEO at BTC Media
  26. Bitcoin either is or is not a store of value. If it is the upside is huge. Email is much bigger than the postal+fax industry. If bitcoin is not a store of value the value will be zero. @DavidFBailey CEO at BTC Media
  27. The bitcoin blockchain is extremely secure. Hacks have happened in the perifery infrastructure. @DavidFBailey CEO at BTC Media
  28. Usdtether has counterparty risk. If the assets backing it are seized the value would drop. @DavidFBailey CEO at BTC Media
  29. If the price of bitcoin booms 10x or 100x, more people will spend more money on mining and the energy consumption of the bitcoin blockchain will go from 0.5% of the world consumption to maybe 10%. What then? @DavidFBailey CEO at BTC Media
  30. Ethereum helps build better business models. This scares incumbents at first. @ethereumJoseph Consensys Founder
  31. R3 is running out of gas. They built a blockchain-like infrastructure. @ethereumJoseph Consensys Founder
  32. Master node in dash is a PoW algorithm. We have instant confirmations. Part of the block rewards go to treasury which can be used to pay for development. Stakeholders can vote on how the rewards get used. @fernando Dash CMO
  33. There is space for many crypto projects. Dash and ethereum can coexist and interoperate. @ethereumJoseph @fernando
  34. Forks are good, it allows for better systems to compete. @ethereumJoseph Consensys Founder
  35. Layer two tech (lightning network) has problems of its own. @fernando Dash CMO
  36. The spark of crypto is still alive and growing thanks to it becoming a reality @fernando Dash CMO
  37. Switzerland is welcoming for ICOs due to their liberal approach to financial innovation @OlgaFeldmeier Smartvalor CEO
  38. In the us there is no clarity on ICOs whereas in switzerland the guidelines are clear and they are welcoming. @ShapeShiftCOO
  39. Different cryptotokens need to be regulated differently we need to understand what they are. @ShapeShiftCOO
  40. Security dealer licenses, banking licenses, money transfer licenses… the regulator needs to simplify @OlgaFeldmeier Smartvalor CEO
  41. The business models for crypto are new. The industry needs to self-regulate with oversight from authorities. Regulator needs to empower the industry players. @OlgaFeldmeier Smartvalor CEO
  42. Digital pictures of cats are not securities. Regulators are struggling to understand how crypto works. @ShapeShiftCOO
  43. Legacy regulation does not catch the bad actors. They are able to evade it anyway. It impedes the good actors and does not allow for fast advance. @ShapeShiftCOO
  44. $300bn market cap on ICO. Bitcoin 38%. @edithyeung Partner at 500Startups
  45. 10 Things to Look for in a Token Project @edithyeung Partner at 500Startups
  46. Avalanche family. New consensus protocol that came out a month ago. Uses montecarlo polling sufficient times for the probability of an incorrect output is insignificant. @el33th4xor Professor at Cornell University
  47. Bitcoin solves the problem of making uncensorable inconfiscable money @ToneVays Derivatives Trader.
  48. fmwill have it’s own crypto. If you have a question fos someone you can offer a payment in crypto in exchange for a response @askfmio
  49. Bitstamp found a regulatory home in luxembourg @nejc_kodric Bitstamp CEO
  50. We started targeting banks. But we realised they were moving too slow so we had to change the strategy. @marcusswanepoel Luno CEO
  51. The biggest struggle is bridging the legacy banking system with the crypto system. Banks are slow and sometimes hostile. @bitstamp @nejc_kodric Bitstamp CEO
  52. In 1980s we went from open outcry markets to electronic markets. Reducing friction and increasing velocity. Next step is the digital age of securities making it 24/7/365 worldwide. Get rid of paper certificates into the blockchain. Tokenize real assets. @APompliano Founder at Morgan Creek Digital Assets
  53. In the future we’ll say: “I bought a share of apple”. The blockchain settlement part will be transparent. @APompliano Founder at Morgan Creek Digital Assets
  54. Ledger is working on Ledgerboard saas solution for enterprise investors who want to manage cryptocurrencies. @EricLarch Ledger CEO
  55. PSD2 is making banks reassess their technical infrastructure. There is an opportunity there. Cabannes @SocieteGenerale Deputy CEO
  56. Psd2 gave banks a kick in the butt. Anyone can come along and access your clients data. This gave banks a wake up call and pushed them to colllaborate with the fintech echosystem @RWandhofer Global Head of Regulatory Strategy at Citigroup
  57. Psd2 increases the surface open to hacker attack. Cabannes @SocieteGenerale Deputy CEO
  58. Psd2 opens the opportunity for collaboration/acquisition of fintechs. This brings innovation to banks. Cabannes @SocieteGenerale Deputy CEO
  59. Banks were reticent to psd2. Now it’s a given so it has been accepted. Psd2 will evolve banking into platforms. There will be competition with winners and losers. This will create a new ecosystem. @deBrouwerEBF Chief Policy Officer at European Banking Federation
  60. Banks have to move from years long IT projects to months, weeks, or even days. Cabannes @SocieteGenerale Deputy CEO
  61. We have to digitize, cleanse, secure, redesign data in big banks @RWandhofer Global Head of Regulatory Strategy at Citigroup
  62. Non-cash payments are growing 82% in india in 2018. India has tripled digital payments in the last three years. @ceouaeexchange CEO at UAE exchange
  63. Millenials care about the experience. Cashless is more intuitive for them. @ceouaeexchange CEO at UAE exchange
  64. In africa most people have cellphones, and few have bank accounts. @ceouaeexchange CEO at UAE exchange
  65. Security issues could pose threats towards cashless societies. Nathan Gill @Verifone Head of Solutions
  66. The simple things that can scale are the things that will proliferate. Nfc works, qr code works. Retina scanning, vein scanning are sexy, but they are more complex. Nathan Gill @Verifone Head of Solutions
  67. Bitcoin fulfills aristotle’s 4 conditions for sound money @maxkeiser CIO Heisenberg Capital
  68. The banking system as we know it will not exist in the next 10-50 years. @maxkeiser CIO Heisenberg Capital
  69. Ireland aspires to be the main tech hub for europe. @campaignforleo Prime minister of Ireland
  70. This year, the Irish central bank will start a fintech innovation hub. @campaignforleo Prime minister of Ireland
  71. Credit karma has 80 million users in usa. Helping millenials find financial instruments @kennethlin Founder of Credit Karma
  72. Through autonomous finance we could give a way for financially illiterate people to make better financial decisions. Similar to the gps helping us reach our destination. @kennethlin Founder of Credit Karma
  73. Worldremit helps the unbanked send and receive money via mobile phone @Ismail_WR Worldremit CEO
  74. 3bn (half of the world) people are invisible to the global financial system. Poor farmers get paid once a year with the harvest and don’t have a way to safely store their money. They need to walk for miles to pay an utility bill to keep the lights on @MichaelSchlein Accion CEO
  75. We use financial tools every day. The underbanked live in a cash world and don’t have access to these tools. @MichaelSchlein Accion CEO
  76. World bank’s report on financial inclusion focuses in bank accounts. There are many underbanked people which created an account and use it less than once a year (they received a government grant and cashed it out, they have an account but don’t use it) @MichaelSchlein Accion CEO
  77. Blockchain can make a difference for cheap remittances without going through the slow and expensive bank based system @Ismail_WR Worldremit CEO
  78. Blockchain is a fundamental new layer to the internet. It enables value exchange, governance and trust. Every fiduciary and record keeping industry will be affected by it. @jerallaire Circle CEO
  79. Accounting. Insurance. Corporate and commercial law. Voting and governance. Record keeping. These will be transformed by the blockchain. @jerallaire Circle CEO
  80. Crypto tokens can be currencies, commodities, and securities. @jerallaire Circle CEO
  81. Stablecoins tether, usd coin, maker, basis can enable settlement without volatility risk @jerallaire Circle CEO
  82. Crypto securities represent some rules based financial contract. @jerallaire Circle CEO
  83. We are at the start of a “tokenisation of everything”. Bonds, fiat currency, insurance contracts, property tokens (land, houses, cars), decision tokens (votes, governance) @jerallaire Circle CEO
  84. To realise the tokenisation of everything we need: 1) mature and scalable public blockchains. 2) fiat backed stablecoins. 3) mature marketplaces 4) regulatory clarity @jerallaire Circle CEO
  85. Fiat stablecoins are mandatory for this to work. We need to control the volatility. @jerallaire Circle CEO
  86. Sharing value globally in the same way we can share information. Cheap. Fast. Easy. Fundamental economic integration. @jerallaire Circle CEO
  87. In China a lot of merchants don’t accept cash any more. Visa, wechat, alipay, but no cash. @AlainFalys Chairman Yoyo Wallet
  88. I just came back from china. I was at a restaurant and some people just stood up and left, some walked to the cashier, some needed the waiter to charge. Everyone paying with the mobile phone. @AlainFalys Chairman Yoyo Wallet
  89. We believe voice is the next frontier. Today 50bn/month searches are done by voice. Voice enables a natural experience. Giulio Montemagno. Director Europe @amazonpay
  90. Mobile topping can be done worldwide in an unregulated way. I can send mobile balance to any phone in the world. This is a workaround for remittances. @MRding Executive Chairman at Ding
  91. Remittances are unfairly expensive. @cwinesLondon Founder at WorldRemit
  92. Gdpr and blockchain don’t work together at all. Blockchain is immutable while gdpr requires user data to allow to be deleted. @DariaRippingale CEO at billpro
  93. Customer demand for cryprocurrency trading has massively exploded. @yoniassia CEO at eToro
  94. Traditional markets are too regulated with too many entry barriers. Crypto markets are the wild west, trial and error. @yoniassia CEO at eToro
  95. You can’t expect to make 40x on all of your investments, right? But you are 27 so you do @yoniassia CEO at eToro
  96. I’m optimistic. In 20 years any financial asset and central bank money will be tokenised in a blockchain. @yoniassia CEO at eToro
  97. Bitcoin has the brand awareness of cocacola. @yoniassia CEO at eToro
  98. Compliance protects the business to avoid risk. Learn to love the audits. Create a team that focuses on responding to audit and loves it. Jacqueline Molnar, Chief Compliance Officer at @WesternUnion
  99. Fintech in europe is very exciting. In europe there is a lot of interest to increase the 5% of online payments. @chughesjohnson COO at Stripe
  100. Governments want to enable startups because they will enable job creation and GDP growth. @chughesjohnson COO at Stripe
  101. Stripe wants to level the playing field allowing any startup to send and receive cross border payments. @chughesjohnson COO at Stripe
  102. Availability of developer resources is according to nielsen the biggest constraint for companies. @chughesjohnson COO at Stripe
  103. Payments is extremely local. Regulation, banking, tradition … Stripe is trying to make it uniform globally for startups to sell worldwide seamlessly @chughesjohnson COO at Stripe
  104. 1) find your product and focus on it full speed 2) If you don’t hire the right talent your company will not grow. 3) tempo. Keep the speed as you grow. @chughesjohnson COO at Stripe
  105. The developer is increasingly the decision maker. The technical decisions you take are integral to your business strategy. Developers should be at the decision making table. If you treat them as a service they will leave. @chughesjohnson COO at Stripe
  106. Before wechatpay and alipay people in china has no credit card. These services leapfrogged and today it’s difficult to be able to pay with cash in china. @edithyeung Partner at 500Startups
  107. Chinese people leave china for hongkong, malta, japan, san francisco… to avoid chinese regulatory blocking and continue their blockchain projects. @edithyeung Partner at 500Startups
  108. We have a process we follow whenever we open up in a new country. If the process does not work we tweak it. We are hungry. @NStoronsky CEO at Revolut.

 

 

 

Euskal Encounter inspiration

Every year late July I attend the Euskal encounter. A meeting of over 8,000 computer enthusiasts who gather to share their passion. For me this is an opportunity to catch up with old friends and discuss our vision of the state of the industry and the future to come. I have seen it evolve over 25 years from the early days when it was a meeting of amiga demoscene fans to today where we have gigabit access to the internet, drone races and training of 10 year old future engineers who are helped to work together to build electric karts and race them.

Every year I obtain a few wisdom pearls that I take back with me and fuel my inspiration. These are the highlights I take with me this time as exciting material for investigation:

  1. Bime is a competitor to Tableau, Qlik, microsoft powerBI, and gooddata in the Business Intelligence space. Setting up a demo is easy and free, and it looks like a powerful front end for end user data analysis.
  2. Tmux is a program which allows you to, just opening one ssh connection to a linux server, split this terminal in as many windows as you like, allowing to split your workspace very flexibly and move fast thanks to keyboard shortcuts. It has similarities with screen.
  3. Terraform allows you to automatically provision your architecture in the cloud. My good friend @ibannieto gave a demo on this for beginners and showed us how easy it is to provision from scratch several machines in the Amazon, Microsoft or Google clouds and how to deploy software on these machines automagically with just a few lines of code.
  4. Awesome is a super lightweight linux window manager that allows to organise your desktop with very powerful keyboard shortcuts and taking up very litte resources. This is ideal when you are running linux on a 10 year old laptop with no resources to spare, as is my case in this occasion.
  5. Google Data Flows is a google ETL service which can manage your data integration needs for real time and batch integration capable of managing massive amounts of data.
  6. Data lakes are a way of storing massive amounts of data without the need for too much structure, allowing for future querying of this data. Hadoop is a data lake which allows for these massive amounts of data distributed amongst thousands of machines to be queried using simple programming models.
  7. Fusion4energy is an exciting european project aiming to bring sustainable energy to the world by using nuclear fusion. Different to nuclear fision, fusion would cause no nuclear residue and would be a revolutionary source of clean energy. Fusion has already been achieved producing Megawatts of energy, the only problem being that it produced less energy than it consumed.
  8. Reveal is a very cool software to create impacting presentations using html. It allows to integrate gifs, video, dynamic content, in a way that microsoft powerpoint can’t.
  9. Drone racing was one of the new surprises of this year’s euskal encounter. The drivers wear headsets with which they see what their flying drone sees and they race at vertiginous speeds through an obstacle race aiming to be the fastest without crashing.
  10. kidskitcar.org was for me the most inspiring of all. Maybe it’s due to my parenthood, but I was moved seeing the different teams of 10 year olds building their electric karts together. I find it crucial to inspire the future generations to take up the challenges of engineering. This way allows them to start easy and small in a fun and collaborative way.

IMG_4911

The future of Virtual Reality

This week I have started using my virtual reality headset and I have been totally blown away by it.

When you put on your headset you are transported to your virtual home, a spacious terrace on a mountaintop on a sunny day, with the birds chirping, the breeze blowing, and a beautiful mountainous landscape. In one wall of your virtual home you can manage your command center from which you can launch any VR application, and see which of your friends are online to play with. The sensation of immersion is perfect, after a few minutes you completely forget that you are not really in the virtual world, and after a few hours of VR, going back to the normal world takes a few seconds of adaptation. The realisation that you have been staring at a wall moving your arms around for hours is confusing.

Traditionally, videogames have always been limited by the screen size, the area of game was necessarily reduced to the size of the screen, hence your avatar in this world was small and so were the rest of creatures in it. This is no longer a limitation. Everything is life-size. You are transported inside the game’s world, for those of you who watched Tron, the movie, this is it. In “Star Trek Bridge Crew”  you are captain of the Aegis space ship navigating through the galaxy. When you look out the window, the immense size of the planets, meteorites and stars is breath taking. In “TheBlu” you are in the bottom of the ocean swimming with medusas, tortoises and whales. The sensation is very real and awe inspiring. In“Richie’s Plank Experience” you have to walk a plank on the top of a skyscraper and you are dared to jump off it. Even if I knew I was in the safety of my room in the real world, I could not command my feet to move. The sensation of vertigo was stronger than my brain’s logic. This is amazing!

One drawback that VR experiences have is that when your body is moving in the VR world but not in the real world, for example if you are piloting a kart, or a spaceship doing barrel rolls, your brain is not capable of understanding why your inner ear is not feeling the pressure of the centrifugal force and you get motion sickness. The sensation is similar to getting seasick. I know first hand.

Imagine working on your computer, not from a chair in the office staring at a couple small screens, but on the top of the Everest on a sunny day, in the bottom of the ocean, or in the center of the galaxy. Imagine your screen is not limited by size, but can be as large as the sky, allowing you have a 360 degree screen where you can leave your different applications. This is what Virtual Desktop does. Currently it has some limitations since the VR headset only has reading-resolution where your head is oriented, not where your eyes are looking. Google is working on a headset with human eye resolution using a trick tracking your eyes to see where you are looking at to maximise resolution in this area.

At any rate of advance in VR technologies, in the next 5-10 years it’s easy to foresee that VR graphics and sound will be indistinguishable from base reality. In this scenario, it would be possible for companies not to need to have physical offices for computer work. Workers could work remotely from their homes, teams would be distributed worldwide with a virtual office in the VR world. This office would be amazingly beautiful, spacious, sunny, with breathtaking landscapes. People would have avatars in this VR world which will allow them to have meetings, work together on projects, explain topics on a whiteboard…

bloomberg-on-oculus

How to mine Ethereum

Mining ethereum is an easy 5 step process:

  1. Get a computer with a powerful GPU or two
  2. Get an ethereum wallet here
  3. Download geth from here and run it like this in a cmd window (because if you have a powerful GPU or two you use it for gaming, hence you have a windows machine)
    1. geth.exe -rpc
  4. Download ethminer from here and run it like this in different cmd window
    1. ethminer.exe -G
    2. The -G parameter instructs ethminer to use your GPU instead of your CPU, which will be orders of magnitude faster at mining.
  5. You are set, you should see something like thisethminer

Now for the bad part. I have two GTX 1080 TI and I am getting around 62 MH/s. This means my computer is capable of taking the Ethereum block we are trying to solve, add it a random string and calculate the hash of it to see if the first ten characters are zeroes and therefore win this block. 62 million times every second. 

62 million times per second sounds like a lot! I should have good chances of winning a block once in a while. Taking into account a new block is generated every 15 seconds, I should have good chances..

The problem is that the Ethereum network has a hash rate capacity of around 60TH/s, this means that the number of hashes the network produces every second are one million times bigger than the hashes my computer produces. This gives me a chance of winning of one in one million every 15 seconds. Taking into account that the reward for winning is 5 ETH (approx 1,150 EUR), I have 1/1,000,000 chance of winning 1,150 EUR every 15 seconds if I leave my computer mining.. this does not sound too promising.

Business Intelligence state of the art

Usually when we need to understand our business’ numbers, we open Microsoft Excel and start applying filters, pivot tables and charts to try to gain insight on what story our data are telling us.

It’s 2017, Isn’t there a better way to do this?

I think it must have been around 2014 when I first heard about Tableau. I immediately liked the tool’s proposal: Easy tool to analyse data in real time without needing to be an expert in SQL, Olap cubes, or any technology. It could take as input for data any number of excel sheets, csv files, or database sources and would immediately allow the user to move the data around, establish relationships between sets of data and start visualising. I’m a visual person, I need to see things in graphical form to understand them, so I really like this concept. In the Gartner Magic Quadrant for Feb 2017, the 3 top players in the BI space by completeness of vision and capacity of execution are Tableau, Microsoft, and Qlik who offer similar solutions for front end analytics.

One of the weak points that I see in Tableau’s proposal is that it only addresses the front end, the data analysis, but leaves the back end, the data model, the origin of data, for you to deal with. In the past I’ve worked on a few data warehousing projects where the immense complexity was in creating a data model which made sense from a business perspective and which could grow with the business’ future needs. Designing the data model wrong was similar to shooting yourself in the foot, as future needs could not be adapted, changes would require refactoring of the whole data model and of the downstream systems which read from this data model and maintenance of this data model would be expensive and complex. In all cases I experienced in the past, creating a data model was similar to creating a monster which would grow in unforeseen and inelegant ways getting more rigid and patchy as time went by.

A good example of a world class level attempt at creating a data model for the capital markets world is  ISDA’s (International Swaps and Derivatives Association) FPML (Financial Products Markup Language). The data model they have birthed is immensely complex, as could not be different when the goal is to be able to map any financial instrument. Any attempt at using FPML as inspiration for your data model promises a long and complex project.

Three weeks ago, I attended Moneyconf. There I saw a company called gooddata which offers an interesting model: They take care of your Business Intelligence needs as a service. You send them your data, they organise it and offer you a Tableau style front end tool to visualise it. This would take care of all data warehousing troubles.. however I have some concerns:

  1. How do they know all the ways you need to structure your data model?
  2. Does this scale well with big volumes of data?
  3. How do they manage confidential data? Imagine I have medical records to manage…
  4. Is the front end tool as complete as Tableau?

 

The BI world is very cool looking. It brings promise of granting better insights and fancy looking charts, but, is the price tag worth it when we can always just grab an excel and start applying filters, pivot tables and charts to understand the story our data are telling us?

bizint