There is a site, on it the input field which, after clicking on it, throws out the menu. The menu is loaded with a JS script.

I need to programmatically click on this input field and after loading the menu with the JS script - extract the html code of the page (with this menu).

I try to reproduce it using HtmlUnit, here's the code

@Test public void testPage() { try (final WebClient webClient = new WebClient(BrowserVersion.CHROME)) { java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(Level.OFF); webClient.getOptions().setUseInsecureSSL(true); webClient.getOptions().setThrowExceptionOnScriptError(false); webClient.getOptions().setThrowExceptionOnFailingStatusCode(false); webClient.getOptions().setCssEnabled(true); webClient.getOptions().setJavaScriptEnabled(true); webClient.getOptions().setRedirectEnabled(true); final HtmlPage page = webClient.getPage("http://olx.ua/"); assertEquals("Доска объявлений OLX, раньше Slando: сайт частных объявлений в Украине - купля/продажа б/у товаров на OLX.ua", page.getTitleText()); webClient.waitForBackgroundJavaScript(5000); HtmlInput input = page.getHtmlElementById("cityField"); input.click(); webClient.waitForBackgroundJavaScript(3000); input.focus(); webClient.waitForBackgroundJavaScript(3000); String htmlCode = page.getWebResponse().getContentAsString(); System.out.println(htmlCode); } catch (MalformedURLException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } }

It seems to do everything right, but in the console I get only a static version of the page, without a menu (and I search for the menu via CTRL + F by class / id in the console).

How can this problem be solved ?

    1 answer 1

    HtmlUnit is easy and convenient, but far from complete. If you have a rather complicated JS, then it is better to use a full-fledged Selenium. There will be easy edits in dependencies and initialization. It will take more memory.

    • With Selenium made everything in 5 minutes, it is very cool! Thank you very much! - Andrey Sibirkin