Author Archives: Miguel Raz Guzmán Macedo

Julia's nothing, missing and NaN – a simple mental model

By: Miguel Raz Guzmán Macedo

Re-posted from: https://miguelraz.github.io/blog/nothingforbeginners/index.html

A quick note for my Julia peeps to grok the difference between NaN, missing and nothing in JuliaLang. I have a few friends on twitter that remind me that the distinction between these concepts is not trivial, but I think I have a good mental model of how to address it and I might as well write it up. Hat tip to Jasmine Hughes for inspiring this post and also sponsoring me on GitHub so I can continue my open source campaing.

A rainy setup

You are running a science experiment where you must measure the amount of rainwater that falls in a given day. The scientific-est thing your advisor has recommend is to setup a RainWater-O-Tron 9000 that collects data every day on how much water fell into a tube that sticks on top of it and reports it back the total at the end of the day. Once the thing is plugged, your machine graciously spits into your data pipeline tool a table that looks something like this:

Days Rain [cm] Status
1 15 OK
2 20 OK
3 10 OK

So far, nothing out of the ordinary. Another humble data gathering expedition to appease the fickle gods of science and grants. The machine kindly records the centimeters of rain collected and its operating status – seems sensible.

You reset the machine and leave for Easter break and leave the robot running for a week, ready to come back and do some proper Science TM once you get the data back.

Ominously, you find the report to say this:

Days Rain [cm] Status
1 12 OK
2 22 OK
3 13 OK
4 OK
5 NO
6 💩 OK
7 18 OK

Clearly something has gone wrong, on days 4-6, but if you think about it carefuly for a second, the Status of each data point gives you some insight into where your data collection could have gone wrong.

  • Days 1-3 went (likely) as expected

  • Day 4 the RainWater-O-Tron recorded it was functional, but you didn't receive the data. You thus know the data for Day 4 is missing.

  • Day 5 the machine wasn't even functional, and thus no data was collected, which means you have nothing as a data entry.

  • Day 6 the machine did record data, but it got garbled somehow, and the result you got is Not a Number/NaN.

  • Day 7 it seems the machine resumed normal operations.

This is the big distinction in how much you know about your data, and the "failure modes" of how it was mis/collected: you get an idea for how to approach its shortcomings based on what you recorded.

  • for the missing data, perhaps the machine ran out of memory from the moment it made the water measurement accurately, but didn't transmit it, or the cable got bitten by some rats and thus you couldn't receive it

  • the nothing data means that perhaps there was a power outage, and your entire apparatus was offline

  • NaN means the internal functioning of the machine got compromised, or something in your calculations is wildly wrong

Of course, these are just narratives for illustrative purposes, but hopefully it can help solidify the distinctions and how these can help you think to solve your problem. Does that mean you must always use these sentinel values in your code or data collection? Not necessarily, but that's for you to decide if these are the right tools.

'Til next time.

Popotismo

By: Miguel Raz Guzmán Macedo

Re-posted from: https://miguelraz.github.io/blog/popotismo/index.html

Hola, me dedico al software libre y este es un pequeño post para poder acuñar una palabra que me falta en el diario. Puedes ser mi sponsor en GitHub para que escriba más cosas como esta y contribuya al software libre y traducciones en español.

Órdenes de magnitud

Cuando uno quiere saber cuánto pesa una elefante, no te preocupas (mucho) por cuánto pesan las moscas que se le paran en la cabeza. Por supuesto que su peso no es 0, pero no contribuyen tanto como para preocuparse. Esta intuición cotidiana se puede formalizar si uno estudia matemáticas, y se les llama "términos de menor órden", y en algún momento del despeje algebraico uno se toma la licencia de tacharlos y seguir campante con el análisis del fenómeno de interés.

En este sentido, es muy útil el poder tener una intuición de los órdenes de magnitud en una cierta discusión antes de aventarse por la borda en favor o en contra de alguna causa. En México, por ejemplo, si alguien quiere tener una discusión seria sobre el ambientalismo, es útil tener en mente los números que más importan (el peso del elefante pues), como el hecho que 100 compañías son responsables del 71% de todos los gases invernadero.

Teniendo en mente ese elefante, no es que uno no debería evitar el consumo de plásticos deshechables gratuitamente, pero hay que darse cuenta que uno está evitando la discusión principal de cómo lidiar con los términos de mayor orden. En el peor de los casos, sobra la santurronería y se desgarran las vestiduras los políticos para castigar a la gente por seguir usando popotes.

A éste fenómeno donde el problema estructural se trata de resolver con recursos de responsabilidad individual le llamo popotismo, en honor a la campaña masiva que se dió súbitamente en México para prohibir el uso de popotes de plástico a nivel individual, y creo que no vale la pena seguir esa línea de argumentación.

Sí, recicla, disminuye tu consumo, eso es bueno y vale la pena, etc; pero a veces la solución es cambiar el sistema y no encontentarse que uno pone un granito de arena en donde ya extrajeron la playa.

El juego es otro.