I don’t have any idea about any undergraduate Physics Olympiad in Bangladesh. But there is undergraduate olympiad for Mathematics every year — National Undergraduate Mathematics Olympiad | Bangladesh Mathematical Society

But, the physics students should not be frustrated, because there are many online international competitions for Undergrad and Post Grad physicists. And it’s a pleasure that many Bangladeshi physics enthusiasts from different universities are now participating and doing very great.

Here’s a shortlist of the competitions I know about:

• The University Physics Competition
• International Olympiad in Theoretical Physics
• Online Physics Brawl
• Physics Cup - Online Physics Competition
• Rudolf Ortvay Competition in Physics
• Online Physics Olympiad
• International Astronomy and Astrophysics Competition

I think you will find many more if you search on the internet.

And here’s a blog-post where these international competitions are briefly discussed in Bangla language - পদার্থবিজ্ঞানভিত্তিক আন্তর্জাতিক অনলাইন প্রতিযোগিতা.

Best of luck!

Update (2019):
Since 2018, the Department of Physics at SUST arranges a contest (sustphysicsc) annually which is inspired by the above-mentioned University Physics Competition. According to the 2019 contest rule, the undergrad students from any institution in the Sylhet division can participate there.