Category Archives: Julia

oneAPI.jl 1.5: Ponte Vecchio support and oneMKL improvements

By: Tim Besard

Re-posted from: https://juliagpu.org/post/2024-05-24-oneapi_1.5/index.html

oneAPI.jl v1.5 is a significant release that brings many new features, from extended hardware support to greatly improved wrappers of the oneMLK math library.

Intel Ponte Vecchio

In oneAPI.jl v1.5 we introduce support for the Intel Ponte Vecchio (PVC) architecture, which empowers the Xe HPC GPUs as found in the Aurora supercomputer:

julia> oneAPI.versioninfo()
Binary dependencies:
- NEO: 24.13.29138+0
- libigc: 1.0.16510+0
- gmmlib: 22.3.18+0
- SPIRV_LLVM_Translator_unified: 0.4.0+0
- SPIRV_Tools: 2023.2.0+0Toolchain:
- Julia: 1.10.3
- LLVM: 15.0.71 driver:
- 00000000-0000-0000-17d2-6b1e010371d2 (v1.3.29138, API v1.3.0)16 devices:
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550
- Intel(R) Data Center GPU Max 1550

Apart from a handful of MKL-related issues, oneAPI.jl is fully functional on PVC, and passes all tests.

oneMLK wrappers

Thanks to the work of @amontoison, oneAPI.jl now provides greatly improved wrappers of the oneMKL library. This includes support for:

  • LAPACK: geqrf(_batched), orgqr(_batched), ormqr, potrf(_batched), potrs(_batched), getrf(_batched), getri(_batched), gebrd, gesvd, syevd, heevd, sygvd, hegvd

  • Sparse arrays: sparse_gemm, sparse_gemv, sparse_symv, sparse_trmv, sparse_trsv, sparse_optimize_gemv, sparse_optimize_trsv

Where possible, these functions are integrated with standard library interfaces, e.g., making it possible to simply call eigen, or to multiply two oneSparseMatrixCSRs.

Minor changes

There have of course been many other changes and improvements in oneAPI.jl v1.5. For a full list, please refer to the release notes, but some highlights include:

  • a new launch configuration heuristic that should generally improve performance;

  • broadcast now preserves the buffer type (host, device, or shared);

  • support for very large arrays that exceed the default device memory limit;

  • several toolchain bumps, with v1.5 using oneAPI 2024.1.0 with driver 24.13.29138.7;

  • minimal support for native Windows (next to WSL, which is fully supported).

DataFrames.jl: avoiding compilation

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2024/05/17/compilation.html

Introduction

Today I want to go back to a topic of performance considerations of using anonymous functions in combination with DataFrames.jl.
I have written about it in the past, but it is an issue that new users often ask about.
In the post I will explain you the problem, its causes, and how to avoid it.

The post was written under Julia 1.10.1 and DataFrames.jl 1.6.1.

Performance issues caused by anonymous functions

Consider the following sequence of DataFrames.jl operations (I am suppressing the output of the operations with ; as it is irrelevant):

julia> using DataFrames

julia> df = DataFrame(x=1:3);

julia> @time select(df, :x => (x -> 2 * x) => :x2);
  0.077134 seconds (73.33 k allocations: 5.010 MiB, 99.28% compilation time)

julia> @time select(df, :x => (x -> 2 * x) => :x2);
  0.013731 seconds (6.30 k allocations: 450.148 KiB, 98.15% compilation time)

julia> @time subset(df, :x => ByRow(x -> x > 1.5));
  0.094046 seconds (91.86 k allocations: 6.219 MiB, 99.29% compilation time)

julia> @time subset(df, :x => ByRow(x -> x > 1.5));
  0.086597 seconds (43.05 k allocations: 2.881 MiB, 42.62% gc time, 99.44% compilation time)

In both select and subset examples I used an anonymous function. In the first case it was x -> 2 * x, and in the second x -> x > 1.5.

What you can notice is that most of the time (even in consecutive calls) is spent in compilation. What is the reason for this?

Let me explain this by example. When we pass the x -> 2 * x function to select, the select function needs to be compiled. Since the select function
is quite complex its compilation time is long.

Why does this happen. The reason is that each time we write x -> 2 * x Julia defines a new anonymous function. Julia compiler does not recognize that it is in fact the same function. Have a look here:

julia> x -> 2 * x
#9 (generic function with 1 method)

julia> x -> 2 * x
#11 (generic function with 1 method)

We can see that we get a different function (one denoted #9 and the other #11) although the definition of the function is identical.

How to solve the compilation issue?

Fortunately, there is a simple way to resolve this problem. Instead of using an anonymous function, just use a named function:

julia> times2(x) = 2 * x
times2 (generic function with 1 method)

julia> @time select(df, :x => times2 => :x2);
  0.013728 seconds (5.63 k allocations: 401.305 KiB, 98.54% compilation time)

julia> @time select(df, :x => times2 => :x2);
  0.000142 seconds (71 allocations: 3.516 KiB)

julia> gt15(x) = x > 1.5
gt15 (generic function with 1 method)

julia> @time subset(df, :x => ByRow(gt15));
  0.041173 seconds (42.64 k allocations: 2.849 MiB, 99.01% compilation time)

julia> @time subset(df, :x => ByRow(gt15));
  0.000165 seconds (120 allocations: 5.648 KiB)

Now you see that consecutive calls are fast and do not cause compilation.

Actually, instead of defining the gt15 function we could just have written >(1.5):

julia> >(1.5)
(::Base.Fix2{typeof(>), Float64}) (generic function with 1 method)

Which defines a functor that works as a named function (so it requires only one compilation):

julia> @time subset(df, :x => ByRow(>(1.5)));
  0.075423 seconds (41.80 k allocations: 2.804 MiB, 99.32% compilation time)

julia> @time subset(df, :x => ByRow(>(1.5)));
  0.000189 seconds (124 allocations: 5.898 KiB)

If you want to learn how functors work in Julia, have a look here.

Conclusions

Today I have presented some simple examples, but I hope that they are useful for new users of DataFrames.jl in helping them to improve the performance of their code.

Building an Interactive Flight Analysis Dashboard with Dash.jl

By: Maja Gwozdz

Re-posted from: https://info.juliahub.com/blog/building-an-interactive-flight-analysis-dashboard-with-dash.jl

Welcome to our exploration of building an interactive web dashboard using Dash.jl, a Julia framework that allows for the creation of highly interactive, web-based data visualizations. This post will guide you through the development of a simple flight analysis dashboard. We will focus on combining interactive components and experimenting with different map projections. You can easily adapt this code to your own project. Let’s get started!

Dashboard Details

The flight analysis dashboard is designed to provide the visualization of flight data. Users can easily interact with the data through various controls like dropdown menus for aircraft type selection, radio buttons for choosing map projections, and dynamic maps that display flight paths and details based on user inputs.

You can download the dataset and the final code.

Our Dash.jl application will look like this:

The dataset is very basic and contains fictional flight details. Here is an excerpt:

Key Features

  • Dynamic Filtering: Users can select specific aircraft types to visualize only the flights corresponding to those types.

  • Interactive Maps: The dashboard uses PlotlyJS to render geographical data interactively, allowing users to explore different views and details of the flights.

  • Responsive Design: Built with the responsive capabilities of Dash.jl, the dashboard adjusts to different user inputs in real-time, providing an engaging user experience.

By the end of this blog, you will have a comprehensive understanding of how to leverage Julia and Dash.jl to build an interactive web application for flight data analysis. Let’s dive into the details and start building our dashboard!

Setting Up the Layout

In this section, we’ll dive into setting up the layout for a flight analysis dashboard using Dash.jl, focusing on how the layout and interactive components are constructed to make data exploration easier.

First, let’s include the relevant packages in the first line of our code:

using Dash, DashHtmlComponents, DashCoreComponents, CSV, DataFrames, PlotlyJS

Converting the dataset

Our dataset is in the CSV format. Let’s turn it into a dataframe to make data manipulation easier:

# Load the dataset
df = CSV.File("path_to_dataset.csv") |> DataFrame

Once we’ve specified the data, let’s add this line to our code:

app = dash()

This line initializes a new Dash application.

Configuring the Application Layout

The layout of the Dash application is where the components that make up the user interface are defined. If you have previous experience with HTML, you’ll find component configuration really intuitive.

app.layout = html_div() do
    [
        html_h1("Flight Analysis"),
        html_div("Dash.jl webinar"),

        dcc_dropdown(
            id="aircraft-type-dropdown",
            options=[
                Dict("label" => "All", "value" => "All");
                [Dict("label" => aircraft_type, "value" => aircraft_type) for aircraft_type in unique(df[!, :AircraftType])]
            ],
            value="All" # Default value
        ),
        dcc_radioitems(
            id="projection-type-selector",
            options=[
                Dict("label" => "Equirectangular", "value" => "equirectangular"),
                Dict("label" => "Orthographic", "value" => "orthographic"),
            ],
            value="equirectangular" # Default value
        ),
        dcc_graph(id="flight-map"),
        html_div(id="flight-details")
    ]
end

The components

We will focus on the interactive components in the next section. At this point, let’s look at the basic building blocks:

  1. Header and Subheader:

    • html_h1("Flight Analysis"): Creates a header (h1 HTML tag) for the title of the dashboard.

    • html_div("Dash.jl webinar"): Adds a division (div HTML tag) with text, serving as a subheader or additional information.

  2. Graph and Details Section:

    • dcc_graph(id="flight-map"): Defines a space for Plotly graphs; in this case, a map that will display flights based on the selected filters.

    • html_div(id="flight-details"): An empty division that can be used to display additional details about specific flights when a user interacts with the map.

  3. dcc components:

    • By integrating these interactive components, users can adjust the flight map display according to their needs. In our example, the dropdown allows for specific data filtering, while the radio buttons enable switching between different visual representations of the geographical data.

Dropdown for Aircraft Type Selection

dcc_dropdown(
   id="aircraft-type-dropdown",
   options=[
       Dict("label" => "All", "value" => "All");
       [Dict("label" => aircraft_type, "value" => aircraft_type) for aircraft_type in unique(df[!, :AircraftType])]
   ],
   value="All" # Default value
)

Code explanation

  • ID: Identifies this dropdown uniquely within the app, allowing it to be used by callbacks.

  • Options: Populated dynamically using a list comprehension that iterates through each unique aircraft type obtained from the dataset. This list is prefixed with an option labeled “All”, which allows users to select all aircraft types.

    • Dict("label" => "All", "value" => "All"): Adds an option for displaying all data.

    • Comprehension: Generates a dictionary for each aircraft type where the label is the human-readable text shown in the dropdown, and value is the corresponding value used in the data.

  • Value: Sets “All” as the default selection so that initially, the dashboard displays data for all aircraft types.

Radio Items for Map Projection Selection

dcc_radioitems(
   id="projection-type-selector",
   options=[
       Dict("label" => "Equirectangular", "value" => "equirectangular"),
       Dict("label" => "Orthographic", "value" => "orthographic"),
   ],
   value="equirectangular" # Default value
)

Code explanation

  • ID: Again, as with the dropdown above, it uniquely identifies the group of radio items within the dashboard.

  • Options: This array contains two dictionaries, each representing a different type of map projection available for displaying the geographical data.

    • Equirectangular: This projection represents a standard map projection used commonly in world maps.

    • Orthographic: Provides a globe-like, spherical view of the Earth, which can be useful for a different visual perspective.

  • Value: The default projection is set to “equirectangular”, so this view is loaded when the dashboard is initially displayed.

Understanding Callbacks

In this part, we explore how callbacks are used in our Dash.jl application to dynamically update a flight map based on user inputs. This functionality is central to interactive data visualization, and leads to a responsive interface that adapts to changes in user preferences.

Setting Up the Callback

callback!(app, Output("flight-map", "figure"),
          [Input("aircraft-type-dropdown", "value"), Input("projection-type-selector", "value")])

As a general tip, try to use meaningful names for inputs, outputs, and figures. Even if your Dash application is concise, this convention will help you avoid simple bugs.

Components of the Callback

  • Target Output: Output("flight-map", "figure") specifies that the output of the callback will be the figure property of the component with the ID “flight-map”. This component is a graph that will display the flights map.

  • Inputs: The callback listens to changes in two inputs:

    • Input("aircraft-type-dropdown", "value") captures changes in the selected aircraft type from the dropdown.

    • Input("projection-type-selector", "value") tracks changes in the selected map projection from the radio items.

The Callback Function

Inside the callback function, the inputs are used to filter and display the data accordingly.

do aircraft_type, selected_projection
    filtered_df = if aircraft_type == "All"
        df
    else
        filter(row -> row.AircraftType == aircraft_type, df)
    end

Data Filtering

  • All Aircraft Types: If “All” is selected, the entire dataset is used.

  • Specific Aircraft Type: If a specific type is chosen, the dataset is filtered to only include flights of that type. This is done using the filter function with a conditional that checks the AircraftType of each row against the selected type.

Creating the Map Visualization

Let’s adjust our map parameters:

map_trace = scattergeo(lat=filtered_df[!, "Latitude"], lon=filtered_df[!, "Longitude"], mode="markers",
                       text=filtered_df[!, "FlightID"], marker=Dict(:size=>10, :color=>"red"))

layout = Layout(title="Flights Map", geo=Dict(:scope=>"world", :showland=>true, :landcolor=>"rgb(243, 243, 243)",
                                               :countrycolor=>"rgb(204, 204, 204)", :showcountries=>true,
                                               :projection=>Dict(:type=>selected_projection)))
  • Map Trace: Defines how the data points (flights) are plotted on the map. Each point is marked by its geographical coordinates with additional styling (red color, size of markers).

  • Layout: Sets up the overall appearance of the map, including map projection, colors, and whether to show country borders and land.

Return the Plot

return PlotlyJS.plot([map_trace], layout)

Almost finished! This line generates the updated map based on the filtered data and layout specifications, and returns it as the new figure for the “flight-map” graph component.

To run the application, navigate to the Julia REPL and type:

include("path_to_your_dash_app.jl")

We hope this post inspires you to consider Julia and Dash.jl for your next data visualization project, unlocking new possibilities in data exploration and interactive web development!