Python – Segmentation fault

เวลาเจอ error ว่า Segmentation fault โดยไม่มีคำอธิบายใดๆ จะไปไม่เป็น

เลยจะให้พ่น error ออกมาด้วย Python faulthandler module

First add the following to the top of your module.

import faulthandler; faulthandler.enable()

Then re-run your program with the faulthandler startup flag.

# pass as an argument
python -Xfaulthandler test.py

# Or as an environment variable.
PYTHONFAULTHANDLER=1 python test.py

ก็จะได้ประมาณนี้

$ /usr/local/bin/python3.7  -Xfaulthandler test.py
Fatal Python error: Segmentation fault

Current thread 0x00007fc9540b6740 (most recent call first):
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 261 in items
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 310 in __len__
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/core/series.py", line 599 in __len__
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 962 in _try_convert_data
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 1172 in <lambda>
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 1152 in _process_converter
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 1172 in _try_convert_types
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 892 in parse
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 777 in _get_object_parser
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 755 in read
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 618 in read_json
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/util/_decorators.py", line 296 in wrapper
  File "/home/jack/.local/lib/python3.7/site-packages/pandas/util/_decorators.py", line 199 in wrapper
  File "test.py", line 4 in <module>
Segmentation fault

ปัญหา Error – lzma

UserWarning: Could not import the lzma module.

/home/ctlfw/.local/lib/python3.7/site-packages/pandas/compat/__init__.py:120: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
  warnings.warn(msg)

ทางแก้ติดตั้ง lzma-dev

For ubuntu: sudo apt-get install liblzma-dev

For centos: sudo yum install -y xz-devel

เข้าไปในโฟลเดอร์ที่ใช้ติดตั้ง Python แล้วรัน

Then sudo configure && sudo make && sudo make install

แปลงจาก .json เป็น .csv ด้วย Python 3.8

แปลงด้วย Pandas

ติดตั้ง Pandas

python -m pip install pandas

แปลงจาก .json เป็น .csv ด้วย Pandas

import pandas as pd
df = pd.read_json (r'test.json')
df.to_csv (r'test.csv', index = None)

แปลงด้วย csv, json

import csv, json
fileInput = r'test.json'
fileOutput = r'test.csv'
inputFile = open(fileInput, 'r', encoding='utf-8') #open json file
outputFile = open(fileOutput, 'w', encoding='utf-8') #load csv file
data = json.load(inputFile) #load json content
inputFile.close() #close the input file
output = csv.writer(outputFile) #create a csv.write
output.writerow(data[0].keys())  # header row
f = True
for row in data:
    output.writerow(row.values()) #values row

แต่บางทีจะมีบรรทัดวางเกินมา ก็ลบบรรทัดว่างทิ้ง

filename_ip = 'test.csv'
filename_op = 'test.csv'
file = open(filename_ip, 'r', encoding='utf-8')
text = file.read()
lines = text.split('\n')
new_text = ''
for i in range(0, len(lines)):
    if (lines[i]!=''):
        new_text += lines[i] + '\n'
f = open(filename_op, 'w', encoding='utf-8')
f.write(new_text)
f.close()

ส่วนวิธีนี้ยังไม่ได้ลอง Convert JSON to CSV in Python – GeeksforGeeks

Python 3.8 วาดกราฟด้วย Graphviz

ติดตั้ง graphviz บน Ubuntu

sudo apt install graphviz

ติดตั้ง graphviz บน Windows เปิด cmd ด้วยสิทธิ administrator

choco install graphviz

dependency-graph

ติดตั้งไพธอนไลบรารี graphviz-0.19

python3.8 -m pip install graphviz

ที่บรรทัดแรกของไฟล์ dependency_graph.py เพิ่ม

#!/usr/bin/python3.8

ทำให้ไฟล์ dependency_graph.py รันได้

chmod 755 dependency_graph.py

หาตัวอย่าง source code ภาษา c (GitHub – pvigier/ecs: A simple and easy to use entity-component-system C++ library) มาไว้ใน folder src แล้วทดลองรัน

./dependency_graph.py src/ out -f png

จะได้ไฟล์ภาพ out.png

usage: dependency_graph.py [-h] [-f {bmp,gif,jpg,png,pdf,svg}] [-v] [-c]
                           folder output

positional arguments:
  folder                Path to the folder to scan
  output                Path of the output file without the extension

optional arguments:
  -h, --help            show this help message and exit
  -f {bmp,gif,jpg,png,pdf,svg}, --format {bmp,gif,jpg,png,pdf,svg}
                        Format of the output
  -v, --view            View the graph
  -c, --cluster         Create a cluster for each subfolder

แต่ถ้ารันแล้วได้ error

'dot' is not recognized as an internal or external command

แสดงว่ายังติดตั้ง graphviz ไม่สำเร็จ หรือ หา path ไม่เจอ Graphviz’s dot tool on Windows

pydeps

ติดตั้ง pydeps

python -m pip install pydeps

ตัวอย่างการเรียกใช้

pydeps pydeps
pydeps pydeps --rankdir TB
pydeps pydeps --rankdir BT

ให้ output เป็นไฟล์ png

pydeps -T png pydeps

Python Regular expression

  • docs.python.org – re — Regular expression operations

ค้นหาตัวอักษรระหว่าง AAA และ ZZZ

import re

text = 'gfgfdAAA1234ZZZuijjk'

try:
    found = re.search('AAA(.+?)ZZZ', text).group(1)
except AttributeError:
    # AAA, ZZZ not found in the original string
    found = '' # apply your error handling
print (found)
# found: 1234
import re

text = 'page-0188482009-20211001-0002.json'

try:
    found = re.search('page-\w+-(.+?)-', text).group(1)
except AttributeError:
    found = ''
print (found)
# found: 20211001

หาชื่อไฟล์ที่อยู่หลัง folder ที่กำหนด

import re

text = 'content/ptn_year=2021/ptn_month=10/ptn_day=01/content-20211001.json'

try:
    found = re.search('ptn_day=\w+/(.*)', text).group(1)
except AttributeError:
    found = ''
print (found)
# found: content-20211001.json

Python3 ติดต่อดาต้าเบส MySQL

Create Connection

import mysql.connector

mydb = mysql.connector.connect(
    host="localhost",
    user="yourusername",
    password="yourpassword"
)

print(mydb)
# <mysql.connector.connection_cext.CMySQLConnection object at 0x000002A56602E948>

หรือระบุ database

import mysql.connector

mydb = mysql.connector.connect(
    host="localhost",
    user="yourusername",
    password="yourpassword",
    database="mydatabase"
)

Creating a Database

import mysql.connector

mydb = mysql.connector.connect(
    host="localhost",
    user="yourusername",
    password="yourpassword"
)

mycursor = mydb.cursor()

mycursor.execute("CREATE DATABASE mydatabase")

Show databases

mycursor = mydb.cursor()

mycursor.execute("SHOW DATABASES")

for x in mycursor:
    print(x)

Creating a Table

mycursor.execute("CREATE TABLE customers (name VARCHAR(255), address VARCHAR(255))")
mycursor.execute("CREATE TABLE customers (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(255), address VARCHAR(255))")

Show table

mycursor.execute("SHOW TABLES")

for x in mycursor:
  print(x)

Alter table

mycursor.execute("ALTER TABLE customers ADD COLUMN id INT AUTO_INCREMENT PRIMARY KEY")

Delete table

mycursor.execute("DROP TABLE customers")

Insert Into Table

sql = "INSERT INTO customers (name, address) VALUES (%s, %s)"
val = ("John", "Highway 21")
mycursor.execute(sql, val)

mydb.commit()

print(mycursor.rowcount, "record inserted.")

Insert Multiple Rows

sql = "INSERT INTO customers (name, address) VALUES (%s, %s)"
val = [
    ('Peter', 'Lowstreet 4'),
    ('Amy', 'Apple st 652'),
    ('Hannah', 'Mountain 21'),
    ('Michael', 'Valley 345'),
    ('Sandy', 'Ocean blvd 2'),
    ('Betty', 'Green Grass 1'),
    ('Richard', 'Sky st 331'),
    ('Susan', 'One way 98'),
    ('Vicky', 'Yellow Garden 2'),
    ('Ben', 'Park Lane 38'),
    ('William', 'Central st 954'),
    ('Chuck', 'Main Road 989'),
    ('Viola', 'Sideway 1633')
]

mycursor.executemany(sql, val)

mydb.commit()

print(mycursor.rowcount, "was inserted.")

Select From a Table

We use the fetchall() method, which fetches all rows from the last executed statement.

mycursor.execute("SELECT * FROM customers")

myresult = mycursor.fetchall()

for x in myresult:
    print(x)

fetchone()

The fetchone() method will return the first row of the result:

mycursor.execute("SELECT * FROM customers")

myresult = mycursor.fetchone()

print(myresult)

					

Case Styles: Camel, Pascal, Snake

The most popular ways to combine words into a single string

  • camelCase
  • PascalCase
  • snake_case
  • kebab-case
Case StylesText
Rawuser login count
Camel CaseuserLoginCount
Pascal CaseUserLoginCount
Snake Caseuser_login_count
Snake Case (All Caps)USER_LOGIN_COUNT
Kebab Caseuser-login-count
Case Styles

Snake case to camel case

def snake_to_camel(text):
    text = ''.join(word.title() for word in text.split('_'))
    return text

st = 'user_login_count'
print(snake_to_camel(st))
# UserLoginCount

Camel case to snake case

  • docs.python.org – re — Regular expression operations
import re

def camel_to_snake(text):
  return re.sub(r'(?<!^)(?=[A-Z])', '_', text).lower()

print(camel_to_snake('userLoginCount'))
print(camel_to_snake('HTTPResponseCodeXYZ'))
# user_login_count
# h_t_t_p_response_code_x_y_z
import re

def camel_to_snake(name):
    name = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name)
    return re.sub('([a-z0-9])([A-Z])', r'\1_\2', name).lower()

def to_snake_case(name):
    name = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name)
    name = re.sub('__([A-Z])', r'_\1', name)
    name = re.sub('([a-z0-9])([A-Z])', r'\1_\2', name)
    return name.lower()

print(camel_to_snake('UserLoginCount'))
print(camel_to_snake('camel2_camel2_case'))
print(camel_to_snake('getHTTPResponseCode'))
print(camel_to_snake('HTTPResponseCodeXYZ'))
# user_login_count
# camel2_camel2_case
# get_http_response_code
# http_response_code_xyz

Python OOP

Create a Class

class MyClass:
    x = 5
print(MyClass.x)
# 5

p1 = MyClass()
print(p1.x)
# 5
class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
    
    def myfunc(self):
        print("Hello my name is " + self.name)

p1 = Person("Jack", 36)
print(p1.name)
print(p1.age)
p1.myfunc()
# Jack
# 36
# Hello my name is Jack

Python Inheritance

class Person:
  def __init__(self, fname, lname):
    self.firstname = fname
    self.lastname = lname

  def printname(self):
    print(self.firstname, self.lastname)

x = Person("John", "Doe")
x.printname()
# John Doe
class Student(Person):
    def __init__(self, fname, lname, year):
        super().__init__(fname, lname)
        self.graduationyear = year

    def welcome(self):
        print("Welcome", self.firstname, self.lastname
              , "to the class of", self.graduationyear)

x = Student("Mike", "Olsen", 2019)
x.printname()
x.welcome()
# Mike Olsen
# Welcome Mike Olsen to the class of 2019




ตัวอย่าง Python

แสดงชื่อไฟล์ในไดเร็กทอรี

เก็บเฉพาะชื่อไฟล์

from os import walk

mypath = 'C:\\'
liFile = []
for (dirpath, dirnames, filenames) in walk(mypath):
    liFile.extend(filenames)
    break # comment this line

for filename in liFile:
    print(filename)
import glob, os
os.chdir(".")
for file in glob.glob("*.*"):
    print(file)

เก็บ full path

from os import walk

mypath = 'C:\\'
filepaths = []
for (dirpath, dirnames, filenames) in walk(mypath):
    for filename in filenames:
        filepaths.append(dirpath + '\\' + filename)
    break # comment this line
for filepath in filepaths:
    print(filepath)

การเขียนไฟล์ text

text = 'abcde'
filename = 'test.txt'
with open(filename, "w", encoding="utf-8") as f:
    f.write(text)

การอ่านไฟล์ text

filename = 'test.txt'
f = open(filename, "r")
text = f.read()
print(text)
f = open(filename, "r", encoding="utf-8")

การอ่านไฟล์ json

import json

filename = 'test.json'
f = open(filename, "r")
response = json.load(f)
print(response)
print(type(response))
print(response.keys())

date

from datetime import date
date.today()
print(type(date.today()))
print(date.today())
# <class 'datetime.date'>
# 2021-10-14

การรวมไฟล์ .csv ที่มี header แบบง่ายๆ

import os
import glob

merge_file = 'people' # folder name
path = "path_to_csv_folder/" + merge_file

os.chdir(path)
all_files = glob.glob("*.*")
# for file in all_files:
#     print(file)

new_text = ''
for f in range(len(all_files)):
    print(f, all_files[f])

    filename = all_files[f]
    file = open(filename, "r", encoding="utf-8")
    text = file.read()
    lines = text.split('\n')

    if f == 0:
        # print('header of the first file')
        new_text = lines[0] + '\n'

    for j in range(1, len(lines)):
        new_text += lines[j] + '\n'

f = open(merge_file + '.csv', "w", encoding="utf-8")
f.write(new_text.strip('\n'))
f.close()

print('Finish')