[BUG] launcher issue using jupyterlab 3.6.3 / sparkmagic 0.21.0

[achovelon]

Describe the bug
Since we migrated jupyter old-notebook version to jupyterlab 3.6.3, we have interface issue when launching sparkmagic. It's working fine and we can create spark session, but we have displayed 2 launchers.

I suspect a compatibility issue with other componant but I couldn't found which one. Is it something you have already experienced ?

To Reproduce
We launch sparkmagic using this command :

%load_ext sparkmagic.magics
%manage_spark

Expected behavior
On the old jupyter the display was fine :

And we were using sparkmagic 0.19.1 and jupyter notebook 6.4.6

Screenshots

Versions:
$ jupyter --version
Selected Jupyter core packages...
IPython : 8.13.2
ipykernel : 6.23.0
ipywidgets : 8.0.6
jupyter_client : 8.2.0
jupyter_core : 5.3.0
jupyter_server : 2.5.0
jupyterlab : 3.6.3
nbclient : 0.7.4
nbconvert : 7.4.0
nbformat : 5.8.0
notebook : 6.5.4
qtconsole : 5.5.1
traitlets : 5.9.0

• SparkMagic : 0.21.0
• Livy (if you know it) : 0.8.0-incubating_2.12 (scala 2.12)
• Spark : 3.2.4

[dosubot]

🤖

Thank you for providing detailed information about the issue you're experiencing. To better understand the problem, could you please provide the following additional details:

1. Are there any error messages in the JupyterLab console when you see the double launcher issue?
2. Could you provide the list of other JupyterLab extensions you have installed? You can get this list by running jupyter labextension list in your terminal.
3. Have you tried to reproduce this issue in a clean environment with only JupyterLab and Sparkmagic installed?

These details will help us to better understand the issue and find a solution.

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[devstein]

Hey @achovelon thanks for making an issue!

In the screenshots, it shows you have a PySpark kernel, which means your shouldn't need those line magics in the first celll

%load_ext sparkmagic.magics
%manage_spark

Can you try it without them and see if everything looks good?

Note: Dosu (the bot) can't read screenshots so ignore it

[achovelon]

Hello,
Thanks for your message. Indeed it works correctly - with or without these two lines.

My point is that after I loaded sparkmagic, when I use %manage_spark it displays in duplicates with the latest version 0.21.0 and it was not the case with the 0.19.1. But I also upgraded jupyter notebook to jupyterhub.

This does not affect our work with Jupyter and Sparkmagic but I wanted to report the display problem. I suspect incompatible versions but I have done a lot of testing and I always come across duplicates displayed using the new version.
Do you have any recommandation on the jupyterhub version to use with sparkmagic 0.21.0 ?

If it's something you couldn't reproduce using jupyterhub 3.2.3 and sparkmagic 0.21.0 I will close the issue.

Thanks for your time.

[devstein]

@achovelon I see. Thank you for flagging. I'll see if we can reproduce and fix

[jshmchenxi]

We've encountered similar issue with jupyterlab 4.2.0 / sparkmagic 0.21.0