getting website source code with c++

[mohammedawni]

hi
assume we have link like this
http://www.cplusplus.com/

i want to get source code of this link using c++ without opening any browser
or how to search string in this page(without opening any browser).

how to do that.
any help.
thank you.

[wizebin]

You can use libcurl to accomplish this task quite easily.

[mohammedawni]

now i have this code and obtain any www.example.com source code
but i want to get the source code of page like that :
http://www.cplusplus.com/forum/windows/17264/

#include <windows.h>
#include <wininet.h>
#include <iostream>
#include <conio.h>
#include <fstream>
#include<string>
#pragma comment (lib, "wininet.lib")

using namespace std;

int main(int argc, char *argv[])
{
	fstream fs_obj;
  fs_obj.open("temp.txt",ios::out | ios::app);  
  HINTERNET hInternet = InternetOpenA("InetURL/1.0", INTERNET_OPEN_TYPE_PRECONFIG, NULL, NULL, 0 );

  HINTERNET hConnection = InternetConnectA( hInternet, "http://www.cplusplus.com/forum/windows/17264/", 80, " "," ", INTERNET_SERVICE_HTTP, 0, 0 ); //enter url here

  HINTERNET hData = HttpOpenRequestA( hConnection, "GET", "/", NULL, NULL, NULL, INTERNET_FLAG_KEEP_CONNECTION, 0 );

  char buf[ 2048 ] ;

  HttpSendRequestA( hData, NULL, 0, NULL, 0 ) ;
  string total;
  DWORD bytesRead = 0 ;
  DWORD totalBytesRead = 0 ;

  while( InternetReadFile( hData, buf, 2000, &bytesRead ) && bytesRead != 0 )
  {
    buf[ bytesRead ] = 0 ; // insert the null terminator.
    total=total+buf;
    printf( "%d bytes read\n", bytesRead ) ;

    totalBytesRead += bytesRead ;
  }

  fs_obj<<total<<"\n--------------------end---------------------\n";
  fs_obj.close();
  printf( "\n\n END -- %d bytes read\n", bytesRead ) ;
  printf( "\n\n END -- %d TOTAL bytes read\n", totalBytesRead ) ;

  cout<<endl<<total<<endl; //it will save source code to (temp.txt) file
  InternetCloseHandle( hData ) ;
  InternetCloseHandle( hConnection ) ;
  InternetCloseHandle( hInternet ) ;
  cin.get();
}

[Thomas1965]

You need to check the return values.

#include <windows.h>
#include <tchar.h>
#include <wininet.h>
#include <iostream>
#include <conio.h>
#include <fstream>
#include<string>

#pragma comment (lib, "wininet.lib")

using namespace std;

void Error (TCHAR *msg)
{
  ::MessageBox (0, msg, NULL, MB_OK | MB_ICONERROR);
  exit (EXIT_FAILURE);
}

int main (int argc, char *argv[])
{
  fstream fs_obj;
  fs_obj.open ("temp.txt", ios::out | ios::app);
  if (!fs_obj)
  {
    Error (_T ("Failed to open stream"));
  }
  HINTERNET hInternet = InternetOpenA ("InetURL/1.0", INTERNET_OPEN_TYPE_PRECONFIG, NULL, NULL, 0);
  if (hInternet == NULL)
    Error (_T ("Error opening internet."));

  HINTERNET hConnection = InternetConnectA (hInternet, "http://www.cplusplus.com/forum/windows/17264/", 80, " ", " ", INTERNET_SERVICE_HTTP, 0, 0); //enter url here
  if (hConnection == NULL)
    Error (_T ("Error opening internet connection"));
  HINTERNET hData = HttpOpenRequestA (hConnection, "GET", "/", NULL, NULL, NULL, INTERNET_FLAG_KEEP_CONNECTION, 0);
  if (hData == NULL)
    Error (_T ("Error opening internet request"));

  char buf[2048];

  BOOL success = HttpSendRequestA (hData, NULL, 0, NULL, 0);
  if (!success)
    Error (_T ("Error sending request"));
  string total;
  DWORD bytesRead = 0;
  DWORD totalBytesRead = 0;

  while (InternetReadFile (hData, buf, 2000, &bytesRead) && bytesRead != 0)
  {
    buf[bytesRead] = 0; // insert the null terminator.
    total = total + buf;
    printf ("%d bytes read\n", bytesRead);

    totalBytesRead += bytesRead;
  }

  fs_obj << total << "\n--------------------end---------------------\n";
  fs_obj.close ();
  printf ("\n\n END -- %d bytes read\n", bytesRead);
  printf ("\n\n END -- %d TOTAL bytes read\n", totalBytesRead);

  cout << endl << total << endl; //it will save source code to (temp.txt) file
  InternetCloseHandle (hData);
  InternetCloseHandle (hConnection);
  InternetCloseHandle (hInternet);
  cin.get ();
}

[mohammedawni]

thank you.
i got it by replacing line 34
to
HINTERNET hData = HttpOpenRequestA (hConnection, "GET", "/forum/windows/", NULL, NULL, NULL, INTERNET_FLAG_KEEP_CONNECTION, 0);

but i can not get http://www.cplusplus.com/user/ source code with the new informaion after login

http://www.cplusplus.com/user/ source code before login==http://www.cplusplus.com/user/ source code afer login

any help

#include <windows.h>
#include <tchar.h>
#include <wininet.h>
#include <iostream>
#include <conio.h>
#include <fstream>
#include<string>

#pragma comment (lib, "wininet.lib")

using namespace std;

void Error (TCHAR *msg)
{
  ::MessageBox (0, msg, NULL, MB_OK | MB_ICONERROR);
  exit (EXIT_FAILURE);
}

int main (int argc, char *argv[])
{
  fstream fs_obj;
  fs_obj.open ("temp.txt", ios::out | ios::app);
  if (!fs_obj)
  {
    Error (_T ("Failed to open stream"));
  }
  HINTERNET hInternet = InternetOpenA ("InetURL/1.0", INTERNET_OPEN_TYPE_PRECONFIG, NULL, NULL, 0);
  if (hInternet == NULL)
    Error (_T ("Error opening internet."));

  HINTERNET hConnection = InternetConnectA (hInternet, "www.cplusplus.com", 80, " ", " ", INTERNET_SERVICE_HTTP, 0, 0); //enter url here
  if (hConnection == NULL)
    Error (_T ("Error opening internet connection"));
  HINTERNET hData = HttpOpenRequestA (hConnection, "GET", "/forum/windows/", NULL, NULL, NULL, INTERNET_FLAG_KEEP_CONNECTION, 0);
  if (hData == NULL)
    Error (_T ("Error opening internet request"));

  char buf[2048];

  BOOL success = HttpSendRequestA (hData, NULL, 0, NULL, 0);
  if (!success)
    Error (_T ("Error sending request"));
  string total;
  DWORD bytesRead = 0;
  DWORD totalBytesRead = 0;

  while (InternetReadFile (hData, buf, 2000, &bytesRead) && bytesRead != 0)
  {
    buf[bytesRead] = 0; // insert the null terminator.
    total = total + buf;
    printf ("%d bytes read\n", bytesRead);

    totalBytesRead += bytesRead;
  }

  fs_obj << total << "\n--------------------end---------------------\n";
  fs_obj.close ();
  printf ("\n\n END -- %d bytes read\n", bytesRead);
  printf ("\n\n END -- %d TOTAL bytes read\n", totalBytesRead);

  cout << endl << total << endl; //it will save source code to (temp.txt) file
  InternetCloseHandle (hData);
  InternetCloseHandle (hConnection);
  InternetCloseHandle (hInternet);
  cin.get ();
}

[Thomas1965]

What exactly does not work ?

When I run it it works, though it takes sometimes quite a bit.

[mohammedawni]

ok
i log in the website.
and request http://www.cplusplus.com/user/pm.cgi source code
but it shows to me in the source code

<title>User sign in: This feature is only available for registered users</title>

c++ program or the server does not understand that i am logged in the website

how to get correct source code.

[Thomas1965]

You can try so send the username and password with InternetConnect.

[mohammedawni]

does not work
any another idea.

[Thomas1965]

c++ program or the server does not understand that i am logged in the website

Since it works with a browser I guess that they use cookies to remember the login status. I will try in the afternoon with the WebBrowser control in C++/CLI to see if it works there.

EDIT

I did a little app in C++/CLI and the WebBrowser control and it works fine. I can send you the project (VS 2013) if you are interested.

[mohammedawni]

I did a little app in C++/CLI and the WebBrowser control and it works fine. I can send you the project (VS 2013) if you are interested.

nice.
send the project to me.

[Thomas1965]

You gat a PM.

[mohammedawni]

Danke
please look at your pm.