Prerequisites and Environment setup<\/h2>\n
This tutorial is carried out in Anaconda Navigator (Python version \u2013 3.8.3) on Windows Operating System. The following packages need to be installed before you continue with the tutorial \u2013<\/p>\n
- \n
- \n
Pandas<\/p>\n<\/li>\n
- \n
NumPy<\/p>\n<\/li>\n
- \n
Scikit-learn<\/p>\n<\/li>\n
- \n
Matplotlib<\/p>\n<\/li>\n
- \n
Seaborn<\/p>\n<\/li>\n
- \n
Tensorflow<\/p>\n<\/li>\n
- \n
Keras<\/p>\n<\/li>\n
- \n
nltk<\/p>\n<\/li>\n
- \n
re<\/p>\n<\/li>\n
- \n
patoolib<\/p>\n<\/li>\n
- \n
urllib<\/p>\n<\/li>\n
- \n
griddb_python<\/p>\n<\/li>\n<\/ol>\n
You can install these packages in Conda\u2019s virtual environment using
conda install package-name<\/code>. In case you are using Python directly via terminal\/command prompt,pip install package-name<\/code> will do the work.<\/p>\nGridDB Installation<\/h3>\n
While loading the dataset, this tutorial will cover two methods \u2013 Using GridDB as well as Using Pandas. To access GridDB using Python, the following packages also need to be installed beforehand:<\/p>\n
- \n
- GridDB C-client<\/a><\/li>\n
- SWIG (Simplified Wrapper and Interface Generator)<\/li>\n
- GridDB Python Client<\/a><\/li>\n<\/ol>\n
1. Dataset Overview<\/h2>\n
The dataset consists of about 40000 articles consisting around equal number of fake as well as real news Most of the news where collected from U.S newspapers and contain news about american politics, world news, news etc.<\/p>\n
https:\/\/www.kaggle.com\/datasets\/clmentbisaillon\/fake-and-real-news-dataset<\/p>\n
2. Importing Required Libraries<\/h2>\n
\n#import griddb_python as griddb\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport urllib.request\nimport patoolib\n\nimport nltk\nimport string\nfrom nltk.corpus import stopwords\nimport re\n\nimport tensorflow as tf\nfrom keras.preprocessing.text import Tokenizer\nfrom tensorflow.keras.utils import to_categorical\nfrom keras.preprocessing.sequence import pad_sequences\n\nfrom sklearn.metrics import classification_report,confusion_matrix,accuracy_score\nfrom sklearn.model_selection import train_test_split\n\nimport warnings\nwarnings.filterwarnings('ignore')\n%matplotlib inline<\/code><\/pre>\n<\/div>\n3. Loading the Dataset<\/h2>\n
Let\u2019s proceed and load the dataset into our notebook.<\/p>\n
3.a Using GridDB<\/h3>\n
Toshiba GridDB is a highly scalable NoSQL database best suited for IoT and Big Data. The foundation of GridDB\u2019s principles is based upon offering a versatile data store that is optimized for IoT, provides high scalability, tuned for high performance, and ensures high reliability.<\/p>\n
To store large amounts of data, a CSV file can be cumbersome. GridDB serves as a perfect alternative as it in open-source and a highly scalable database. GridDB is a scalable, in-memory, NoSQL database which makes it easier for you to store large amounts of data. If you are new to GridDB, a tutorial on reading and writing to GridDB<\/a> can be useful.<\/p>\n
Assuming that you have already set up your database, we will now write the SQL query in python to load our dataset.<\/p>\n
\nfactory = griddb.StoreFactory.get_instance()\n\n# Initialize the GridDB container (enter your database credentials)\ntry:\n gridstore = factory.get_store(host=host_name, port=your_port, \n cluster_name=cluster_name, username=admin, \n password=admin)\n\n info = griddb.ContainerInfo(\"false_news\",\n [[\"title\", griddb.Type.STRING],[\"text\", griddb.Type.STRING],[\"subject\", griddb.Type.STRING],\n [\"date\", griddb.Type.TIMESTAMP],\n griddb.ContainerType.COLLECTION, True)\n cont = gridstore.put_container(info) \n data = pd.read_csv(\"False.csv\")\n #Add data\n for i in range(len(data)):\n ret = cont.put(data.iloc[i, :])\n print(\"Data added successfully\")\n \ntry:\n gridstore = factory.get_store(host=host_name, port=your_port, \n cluster_name=cluster_name, username=admin, \n password=admin)\n\n info = griddb.ContainerInfo(\"true_news\",\n [[\"title\", griddb.Type.STRING],[\"text\", griddb.Type.STRING],[\"subject\", griddb.Type.STRING],\n [\"date\", griddb.Type.TIMESTAMP],\n griddb.ContainerType.COLLECTION, True)\n cont = gridstore.put_container(info) \n data = pd.read_csv(\"True.csv\")\n #Add data\n for i in range(len(data)):\n ret = cont.put(data.iloc[i, :])\n print(\"Data added successfully\")<\/code><\/pre>\n<\/div>\nThe read_sql_query function offered by the pandas library converts the data fetched into a panda data frame to make it easy for the user to work.<\/p>\n
\nsql_statement1 = ('SELECT * FROM false_news')\nfalse = pd.read_sql_query(sql_statement, cont)<\/code><\/pre>\n<\/div>\n\nsql_statement2 = ('SELECT * FROM true_news')\ntrue = pd.read_sql_query(sql_statement, cont)<\/code><\/pre>\n<\/div>\nNote that the
cont<\/code> variable has the container information where our data is stored. Replace thecredit_card_dataset<\/code> with the name of your container. More info can be found in this tutorial reading and writing to GridDB<\/a>.<\/p>\nWhen it comes to IoT and Big Data use cases, GridDB clearly stands out among other databases in the Relational and NoSQL space. Overall, GridDB offers multiple reliability features for mission-critical applications that require high availability and data retention.<\/p>\n
3.b Using pandas read_csv<\/h3>\n
We can also use Pandas’
read_csv<\/code> function to load our data. Both of the above methods will lead to the same output as the data is loaded in the form of a pandas dataframe using either of the methods.<\/p>\n\ntrue = pd.read_csv(\"True.csv\")\nfalse = pd.read_csv(\"Fake.csv\")<\/code><\/pre>\n<\/div>\n4. Data Cleaning and Preprocessing<\/h2>\n
\ntrue['label'] = 1\nfalse['label'] = 0<\/code><\/pre>\n<\/div>\nCombining the two datasets into one and adding the columns text and title into one column.<\/p>\n
\nnews = pd.concat([true,false]) \nnews['text'] = news['text'] + \" \" + news['title']\ndf=news.drop([\"date\",\"title\",\"subject\"],axis=1)<\/code><\/pre>\n<\/div>\n\nsns.countplot(x=\"label\", data=news);\nplt.show()<\/code><\/pre>\n<\/div>\n
- GridDB C-client<\/a><\/li>\n
![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2022\/06\/output_25_0.png\")
<\/a><\/p>\n