Building an Interactive Flight Analysis Dashboard with Dash.jl

By: Maja Gwozdz

Re-posted from: https://info.juliahub.com/blog/building-an-interactive-flight-analysis-dashboard-with-dash.jl

Welcome to our exploration of building an interactive web dashboard using Dash.jl, a Julia framework that allows for the creation of highly interactive, web-based data visualizations. This post will guide you through the development of a simple flight analysis dashboard. We will focus on combining interactive components and experimenting with different map projections. You can easily adapt this code to your own project. Let’s get started!

Dashboard Details

The flight analysis dashboard is designed to provide the visualization of flight data. Users can easily interact with the data through various controls like dropdown menus for aircraft type selection, radio buttons for choosing map projections, and dynamic maps that display flight paths and details based on user inputs.

You can download the dataset and the final code.

Our Dash.jl application will look like this:

The dataset is very basic and contains fictional flight details. Here is an excerpt:

Key Features

  • Dynamic Filtering: Users can select specific aircraft types to visualize only the flights corresponding to those types.

  • Interactive Maps: The dashboard uses PlotlyJS to render geographical data interactively, allowing users to explore different views and details of the flights.

  • Responsive Design: Built with the responsive capabilities of Dash.jl, the dashboard adjusts to different user inputs in real-time, providing an engaging user experience.

By the end of this blog, you will have a comprehensive understanding of how to leverage Julia and Dash.jl to build an interactive web application for flight data analysis. Let’s dive into the details and start building our dashboard!

Setting Up the Layout

In this section, we’ll dive into setting up the layout for a flight analysis dashboard using Dash.jl, focusing on how the layout and interactive components are constructed to make data exploration easier.

First, let’s include the relevant packages in the first line of our code:

using Dash, DashHtmlComponents, DashCoreComponents, CSV, DataFrames, PlotlyJS

Converting the dataset

Our dataset is in the CSV format. Let’s turn it into a dataframe to make data manipulation easier:

# Load the dataset
df = CSV.File("path_to_dataset.csv") |> DataFrame

Once we’ve specified the data, let’s add this line to our code:

app = dash()

This line initializes a new Dash application.

Configuring the Application Layout

The layout of the Dash application is where the components that make up the user interface are defined. If you have previous experience with HTML, you’ll find component configuration really intuitive.

app.layout = html_div() do
    [
        html_h1("Flight Analysis"),
        html_div("Dash.jl webinar"),

        dcc_dropdown(
            id="aircraft-type-dropdown",
            options=[
                Dict("label" => "All", "value" => "All");
                [Dict("label" => aircraft_type, "value" => aircraft_type) for aircraft_type in unique(df[!, :AircraftType])]
            ],
            value="All" # Default value
        ),
        dcc_radioitems(
            id="projection-type-selector",
            options=[
                Dict("label" => "Equirectangular", "value" => "equirectangular"),
                Dict("label" => "Orthographic", "value" => "orthographic"),
            ],
            value="equirectangular" # Default value
        ),
        dcc_graph(id="flight-map"),
        html_div(id="flight-details")
    ]
end

The components

We will focus on the interactive components in the next section. At this point, let’s look at the basic building blocks:

  1. Header and Subheader:

    • html_h1("Flight Analysis"): Creates a header (h1 HTML tag) for the title of the dashboard.

    • html_div("Dash.jl webinar"): Adds a division (div HTML tag) with text, serving as a subheader or additional information.

  2. Graph and Details Section:

    • dcc_graph(id="flight-map"): Defines a space for Plotly graphs; in this case, a map that will display flights based on the selected filters.

    • html_div(id="flight-details"): An empty division that can be used to display additional details about specific flights when a user interacts with the map.

  3. dcc components:

    • By integrating these interactive components, users can adjust the flight map display according to their needs. In our example, the dropdown allows for specific data filtering, while the radio buttons enable switching between different visual representations of the geographical data.

Dropdown for Aircraft Type Selection

dcc_dropdown(
   id="aircraft-type-dropdown",
   options=[
       Dict("label" => "All", "value" => "All");
       [Dict("label" => aircraft_type, "value" => aircraft_type) for aircraft_type in unique(df[!, :AircraftType])]
   ],
   value="All" # Default value
)

Code explanation

  • ID: Identifies this dropdown uniquely within the app, allowing it to be used by callbacks.

  • Options: Populated dynamically using a list comprehension that iterates through each unique aircraft type obtained from the dataset. This list is prefixed with an option labeled “All”, which allows users to select all aircraft types.

    • Dict("label" => "All", "value" => "All"): Adds an option for displaying all data.

    • Comprehension: Generates a dictionary for each aircraft type where the label is the human-readable text shown in the dropdown, and value is the corresponding value used in the data.

  • Value: Sets “All” as the default selection so that initially, the dashboard displays data for all aircraft types.

Radio Items for Map Projection Selection

dcc_radioitems(
   id="projection-type-selector",
   options=[
       Dict("label" => "Equirectangular", "value" => "equirectangular"),
       Dict("label" => "Orthographic", "value" => "orthographic"),
   ],
   value="equirectangular" # Default value
)

Code explanation

  • ID: Again, as with the dropdown above, it uniquely identifies the group of radio items within the dashboard.

  • Options: This array contains two dictionaries, each representing a different type of map projection available for displaying the geographical data.

    • Equirectangular: This projection represents a standard map projection used commonly in world maps.

    • Orthographic: Provides a globe-like, spherical view of the Earth, which can be useful for a different visual perspective.

  • Value: The default projection is set to “equirectangular”, so this view is loaded when the dashboard is initially displayed.

Understanding Callbacks

In this part, we explore how callbacks are used in our Dash.jl application to dynamically update a flight map based on user inputs. This functionality is central to interactive data visualization, and leads to a responsive interface that adapts to changes in user preferences.

Setting Up the Callback

callback!(app, Output("flight-map", "figure"),
          [Input("aircraft-type-dropdown", "value"), Input("projection-type-selector", "value")])

As a general tip, try to use meaningful names for inputs, outputs, and figures. Even if your Dash application is concise, this convention will help you avoid simple bugs.

Components of the Callback

  • Target Output: Output("flight-map", "figure") specifies that the output of the callback will be the figure property of the component with the ID “flight-map”. This component is a graph that will display the flights map.

  • Inputs: The callback listens to changes in two inputs:

    • Input("aircraft-type-dropdown", "value") captures changes in the selected aircraft type from the dropdown.

    • Input("projection-type-selector", "value") tracks changes in the selected map projection from the radio items.

The Callback Function

Inside the callback function, the inputs are used to filter and display the data accordingly.

do aircraft_type, selected_projection
    filtered_df = if aircraft_type == "All"
        df
    else
        filter(row -> row.AircraftType == aircraft_type, df)
    end

Data Filtering

  • All Aircraft Types: If “All” is selected, the entire dataset is used.

  • Specific Aircraft Type: If a specific type is chosen, the dataset is filtered to only include flights of that type. This is done using the filter function with a conditional that checks the AircraftType of each row against the selected type.

Creating the Map Visualization

Let’s adjust our map parameters:

map_trace = scattergeo(lat=filtered_df[!, "Latitude"], lon=filtered_df[!, "Longitude"], mode="markers",
                       text=filtered_df[!, "FlightID"], marker=Dict(:size=>10, :color=>"red"))

layout = Layout(title="Flights Map", geo=Dict(:scope=>"world", :showland=>true, :landcolor=>"rgb(243, 243, 243)",
                                               :countrycolor=>"rgb(204, 204, 204)", :showcountries=>true,
                                               :projection=>Dict(:type=>selected_projection)))
  • Map Trace: Defines how the data points (flights) are plotted on the map. Each point is marked by its geographical coordinates with additional styling (red color, size of markers).

  • Layout: Sets up the overall appearance of the map, including map projection, colors, and whether to show country borders and land.

Return the Plot

return PlotlyJS.plot([map_trace], layout)

Almost finished! This line generates the updated map based on the filtered data and layout specifications, and returns it as the new figure for the “flight-map” graph component.

To run the application, navigate to the Julia REPL and type:

include("path_to_your_dash_app.jl")

We hope this post inspires you to consider Julia and Dash.jl for your next data visualization project, unlocking new possibilities in data exploration and interactive web development!

The future of AI agents with Yohei Nakajima

By: Logan Kilpatrick

Re-posted from: https://logankilpatrick.medium.com/the-future-of-ai-agents-with-yohei-nakajima-2602e32a4765?source=rss-2c8aac9051d3------2

Delving into AI agents and where we are going next

The future is going to be full of AI agents, but there are still a lot of open questions on how to get there & what that world will look like. I had the chance to sit down with one of the deepest thinkers in the world of AI agents, Yohei Nakajima. If you want to check out the video of our conversion, you can watch it on YouTube:

Where are we today?

There has been a lot of talk of agents over the last year since the initial viral explosion of HustleGPT, where the creator famously told the chatbot system that it had $100 and asked it to try and help him make money for his startup.

Since then, the conversation and interest around agents has not stopped, despite there being a shockingly low number of successful agent deployments. Even as someone who is really interested in AI and has tried many of the agent tools, I still have a grand total of zero agents actually running in production right now helping me (which is pretty disappointing).

Despite the lack of large scale deployments, companies are still investing heavily in the space as it is widely assumed this is the application of LLMs that will end up providing the most value. I have been looking more and more into Zapier as the potential launching point for large scale agent deployments. Most of the initial challenge with agent platforms is they don’t actually hook up to all the things you need them too. They much support Gmail but not Outlook, etc. But Zapier already does the dirty work of connecting with the worlds tools which gets me excited about the prospect this could work out as a tool.

Why haven’t AI agents taken off yet?

To understand why agents have not taken off, you need to really understand the flow that autonomous agents take when solving tasks. I talked about this in depth when I explored what agents were in another post from earlier last year. The TLDR is that current agents typical use the LLM system itself as the planning mechanism for the agent. In many cases, this is sufficient to solve a simple task, but as anyone who uses LLMs frequently knows, the limitations for these planners are very real.

Simply put, current LLMs lack sufficient reasoning capabilities to really solve problems without human input. I am hopeful this will change in the future with forthcoming new models, but it might also be that we need to move the planning capabilities to more deterministic systems that are not controlled by LLMs. You could imagine a world where we also fine-tune LLMs to specifically perform the planning task, and potentially fine-tune other LLMs to do the debugging task in cases where the models get stuck.

Image by Simform

Beyond the model limitations, the other challenge is tooling. Likely the closest thing to a widely used LLM agent framework is the OpenAI Assistants API. However, it lacks many of the true agentic features that you would need to really build and autonomous agent in production. Companies like https://www.agentops.ai/ and https://e2b.dev are taking a stab at trying to provide a different layer of tooling / infra to help developers building agents, but these tools have not gained widespread adoption.

Where are we going from here?

The agent experience that gets me excited is the one that is spun up in the background for me and just automates away some task / workflow I used to do manually. It still feels like we are a very long way away from this, but many companies are trying this using browser automation. In those workflows, you can perform a task once and the agent will learn how to mimic the workflow in the browser and then do it for you on demand. This could be one possible way to decrease the friction in making agents work at scale.

Another innovation will certainly be at the model layer. Increased reasoning / planning capabilities, while coupled with increased safety risks, present the likeliest path to improved adoption of agents. Some models like Cohere’s Command R model are being optimized for tool use which is a common pattern for agents to do the things they need. It is not clear yet if these workflows will require custom made models, my guess is that general purpose reasoning models will perform the best in the long term but the short term will be won by tool use tailored models.

Julia for Data Analysis Strikes Back

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2024/05/10/jda.html

Introduction

This week I got a nice little surprise in my office. A year after my Julia for Data Analysis
book has been published I got a package with a set of printed versions of its Korean translation
데이터 분석을 위한 줄리아. It was really a nice experience and I hope that Julia users from Korea will like it.

데이터 분석을 위한 줄리아

Therefore, for today, I decided to discuss a functionality that is little known, but often quite useful.
It is related to adding conditional columns to a data frame.

The post was written under Julia 1.10.1, DataFrames.jl 1.6.1, and DataFramesMeta.jl 0.15.2.

The problem

Assume you have the following data frame:

julia> using DataFrames

julia> df = DataFrame(x=-2.0:0.5:2.0)
9×1 DataFrame
 Row │ x
     │ Float64
─────┼─────────
   1 │    -2.0
   2 │    -1.5
   3 │    -1.0
   4 │    -0.5
   5 │     0.0
   6 │     0.5
   7 │     1.0
   8 │     1.5
   9 │     2.0

Now we want to add a second column to this data frame that contains a square root of column "x".

A basic approach fails:

julia> df.sqrtx = sqrt.(df.x)
ERROR: DomainError with -2.0:
sqrt was called with a negative real argument but will only return a complex result if called with a complex argument. Try sqrt(Complex(x)).

The reason is that we cannot normally take a square root of a negative number.

We can perform a conditional processing for example like this:

julia> df.sqrtx = (x -> x < 0.0 ? missing : sqrt(x)).(df.x)
9-element Vector{Union{Missing, Float64}}:
  missing
  missing
  missing
  missing
 0.0
 0.7071067811865476
 1.0
 1.224744871391589
 1.4142135623730951

julia> df
9×2 DataFrame
 Row │ x        sqrtx
     │ Float64  Float64?
─────┼─────────────────────────
   1 │    -2.0  missing
   2 │    -1.5  missing
   3 │    -1.0  missing
   4 │    -0.5  missing
   5 │     0.0        0.0
   6 │     0.5        0.707107
   7 │     1.0        1.0
   8 │     1.5        1.22474
   9 │     2.0        1.41421

but I do not find this approach very readable (especially from the perspective of a beginner).

The alternative that I prefer is to work with a view of the source data frame. Let us first create such a view that contains all columns of the original data frame, but only rows in which column "x" is non-negative:

julia> dfv = filter(:x => >=(0.0), df, view=true)
5×2 SubDataFrame
 Row │ x        sqrtx
     │ Float64  Float64?
─────┼───────────────────
   1 │     0.0  0.0
   2 │     0.5  0.707107
   3 │     1.0  1.0
   4 │     1.5  1.22474
   5 │     2.0  1.41421

Now, we can add a column to such a view by using a plain sqrt function without any decorations:

julia> dfv.sqrtx2 = sqrt.(dfv.x)
5-element Vector{Float64}:
 0.0
 0.7071067811865476
 1.0
 1.224744871391589
 1.4142135623730951

julia> dfv
5×3 SubDataFrame
 Row │ x        sqrtx     sqrtx2
     │ Float64  Float64?  Float64?
─────┼─────────────────────────────
   1 │     0.0  0.0       0.0
   2 │     0.5  0.707107  0.707107
   3 │     1.0  1.0       1.0
   4 │     1.5  1.22474   1.22474
   5 │     2.0  1.41421   1.41421

julia> df
9×3 DataFrame
 Row │ x        sqrtx           sqrtx2
     │ Float64  Float64?        Float64?
─────┼─────────────────────────────────────────
   1 │    -2.0  missing         missing
   2 │    -1.5  missing         missing
   3 │    -1.0  missing         missing
   4 │    -0.5  missing         missing
   5 │     0.0        0.0             0.0
   6 │     0.5        0.707107        0.707107
   7 │     1.0        1.0             1.0
   8 │     1.5        1.22474         1.22474
   9 │     2.0        1.41421         1.41421

Note that both dfv and df are updated as expected. The filtered-out rows get missing values.

It is important to highlight that this functionality works if the view (SubDataFrame) was created using all columns of the source data frame (like is done in the case of our filter call above).
The reason for this restriction is that if view contained some subset of columns the operation of adding a column would be unsafe (there would be a risk of accidental and unwanted overwrite of a column present in the source data frame that was not included in the view).

This functionality is especially nice in combination with DataFramesMeta.jl, just have a look:

julia> @chain df begin
           @rsubset(:x >= 0; view=true)
           @rtransform!(:sqrtx3 = sqrt(:x))
           parent
       end
9×4 DataFrame
 Row │ x        sqrtx           sqrtx2          sqrtx3
     │ Float64  Float64?        Float64?        Float64?
─────┼─────────────────────────────────────────────────────────
   1 │    -2.0  missing         missing         missing
   2 │    -1.5  missing         missing         missing
   3 │    -1.0  missing         missing         missing
   4 │    -0.5  missing         missing         missing
   5 │     0.0        0.0             0.0             0.0
   6 │     0.5        0.707107        0.707107        0.707107
   7 │     1.0        1.0             1.0             1.0
   8 │     1.5        1.22474         1.22474         1.22474
   9 │     2.0        1.41421         1.41421         1.41421

In the code above I used parent in the last step to recover the source df.

As a final comment note that an alternative in DataFramesMeta.jl is to just use a plain @rtransform! macro:

julia> @rtransform!(df, :sqrtx4 = :x < 0 ? missing : sqrt(:x))
9×5 DataFrame
 Row │ x        sqrtx           sqrtx2          sqrtx3          sqrtx4
     │ Float64  Float64?        Float64?        Float64?        Float64?
─────┼─────────────────────────────────────────────────────────────────────────
   1 │    -2.0  missing         missing         missing         missing
   2 │    -1.5  missing         missing         missing         missing
   3 │    -1.0  missing         missing         missing         missing
   4 │    -0.5  missing         missing         missing         missing
   5 │     0.0        0.0             0.0             0.0             0.0
   6 │     0.5        0.707107        0.707107        0.707107        0.707107
   7 │     1.0        1.0             1.0             1.0             1.0
   8 │     1.5        1.22474         1.22474         1.22474         1.22474
   9 │     2.0        1.41421         1.41421         1.41421         1.41421

In this case it also quite clean.

Conclusions

I am really happy that we have a Korean version of Julia for Data Analysis.

I hope that the example transformations I have shown today were useful and improved your knowledge of DataFrames.jl and DataFramesMeta.jl packages.